Hello Jeff,
A while ago I noticed that you are using JOCL in Encog (and I linked to your introductory article on CodeProject - thanks for that 
). It’s good to see JOCL applied in such a great project. Any feedback and suggestions resulting from your practical experience with JOCL would be highly appreciated.
Today and during the weekend I won’t have much time, but will continue with git, svn2git and maven on thursday, so hopefully the setup may be finished next week.
bye
Marco
So, after installing Maven and GIT, creating a GitHub account, downloading and compiling the NAR Plugin (and ignoring the error messages), installing Ruby, RubyGems and SVN2GIT, I’m annoyed: The SVN2GIT conversion is plainly not working with a local repository. Apart from that, I’m sure that hardly anyone will be willing to undergo a similar procedure when the alternative is to unpack a simple ZIP and use a single JAR. Seriously, this is trivial, and when something trivial is made that complicated, something went terribly wrong. Maybe I’ll have another look at this next week -_-
svn2git uses git svn internally, which should support URLs like file://local_svn_repo.
once the nar plugin is available on maven, a dev (that uses maven) wanting to use your lib would have a simpler live then going to your site, downloading a zip and extracting a jar. He would rather search for OpenCL or JOCL in his IDE integrated library search thing, and double-click the jocl.org entry in the list.
The conventional/current way of using the lib will still be possible (you could even link to the jar file in he maven repo, as first or mirror download location).
if it is in maven, it should also make the task of supplying native libraries for more exotic platforms more easily for the users, as nar should figure out all the compiler specific stuff.
(sorry for the italic text, i somehow could not turn it off in my browser)
Of course, once everything is set up properly, it **should ** be easier for those who use Maven. I’m just not sure if this will really be the case (considering the version management, and specific details like the ‚include path‘, which presumably still has to be adjusted by manually editing an XML file). Maven seems to have a tremendous internal complexity, which is supposed to be hidden and „…you just have to run ‚mvn‘“ - but if something does not work with the first shot, you’re lost - completely.
I also tried to use the git-svn command instead of svn2git, but it spills some useless error messages which do not say more than „There’s something wrong“ :twisted: However, if it is possible, I will somehow manage to do it, in doubt just to avoid feeling toooo stupid 
At least a small status report, and maybe as a small hint for those trying to convert a local SVN directory (on Windows) into a GIT repository using svn2git: I have been using VisualSVN Server, and it seemed to be necessary to explicitly switch from „Windows Authentification“ to „SVN Authentification“ in the Repository properties of VisualSVN. Additionally, it seems that it’s really not possible to use a local repository using the „file://“ protocol - but the command
svn2git --authors authorsFile.txt https://theComputerName:8443/svn/ProjectName
it finally worked to export the Repository to GIT. At least, partially - it seems that it is necessary to re-run this command several times, to get all branches and tags (otherwise it seems to stop after the first branch). I’ll have to play around with this a little, and see how I can remove the parts of the history which are not intended for the public
I wonder whether it’s so difficult to write a nice UI for simple administrative tasks on a SVN- or GIT repository. This command line interface reminds me of the early 90’s … -_-
And another status report concerning Maven: I finally got it running locally, based on the POM that you (hoijui) uploaded.
To get it running, I had to mess around and add things like
<project xmlns="http://maven.apache.org/POM/4.0.0"
<build>
<plugins>
<plugin>
<configuration>
**
<linker>
<options>
<option>/LIBPATH:C:/Programme/Microsoft SDKs/Windows/v6.0A/Lib</option>
<option>/LIBPATH:C:\Programme\Microsoft Visual Studio 8\VC\lib</option>
<option>C:/Programme/ATI Stream/lib/x86/OpenCL.lib</option>
</options>
</linker>
<cpp>
<includePaths>
<includePath>C:\Programme\ATI Stream\include\CL</includePath>
<includePath>C:/Programme/Microsoft Visual Studio 8/VC/include</includePath>
<includePath>C:/Programme/Microsoft SDKs/Windows/v6.0A/Include</includePath>
</includePaths>
</cpp>
**
</configuration>
</plugin>
</plugins>
</build>
</project>
into the POM.
You mentioned that …
"a dev … would rather search for OpenCL or JOCL in his IDE integrated library search thing, and double-click the jocl.org entry in the list."
And then he would open a console and edit the POM.XML (probably, using Emacs or VI) to replace the include- and library directories Of course, this does not even remotely make sense -_- Maybe this can be avoided somehow…
EDIT: By the way, concerning the conversion of SVN to GIT: In order to separate the projects properly, and to avoid exposing the non-public parts of my earlier commits, I’ll probably integrate the current version and keep the history only in my local SVN. I’m not yet really convinced to switch to GIT anyhow.
EDIT2: It was intended as a joke, but I just noticed that VIM is used for --amend’ing commits. This is wrong on so many levels… 
a dev using your lib should of course not have to compile the native part at all. keep in mind i never put a lib with native parts onto maven myself, but from what i read in the nar plugin doc, you would make one bundle or multiple ones with the native parts compiled, possibly as separate packages on maven.
for devs actually working on the library, adding patches, there are several ways of solving it:
git uses makes use of a text editor for some actions, it therefore uses the one you defined as system standard through the EDITOR env var, i think. if it is not so, there is an other way to change the editor.
as it sounds really to be quite complex for you to get it to git (guess that is mostly related to git2svn and git svn being largely untested for in the MS world), it might be good to forget that history and have a fresh start. maybe it woudl be good though, to have at least one commit for each release you had so far. will be a bit boring to make, but should be easy/problem-less. if you want me to do it, you had to give me your authors.txt.
EDIT: Huh, that was close Didn’t see the last post…
Does anybody have an idea how the necessary linker options could be sensibly integrated into the POM? According to Maven – POM Reference at least the include path (and possibly the OpenCL lib path) may be added via an environment variable. But the Visual Studio Linker needs some options to either add the path for the „standard library“ - or options to omit these „standard libraries“, as far as this is possible.
OK, that answers some of my questions.
When you say that the native lib should be pre-compiled and integrated into a dedicated package, I hardly see a reason for adding the nar-plugin info into the POM. Do you think it would make sense to have two different POMs, one for the pure Java user, and one („advanced“ ) for those who also want to compile the native part on their own? For the latter, I think that a mixture of the above mentioned Environment variables, and possibly an XML file could solve most of the problems. As far as I understood, it should be possible to define parts of the POM specifically for the respective target operating system, where the POM is executed.
I think there are three sources for difficulties in the process of getting the code into GIT:
Concerning the migration itself: There are currently several JOCL-related projects in my local SVN:
I’ll try to create the GIT repository containing the commits corresponding to the previous releases by next week.
ehh nice! 
sounds like big progress there!
how the necessary linker options could be sensibly integrated into the POM
Maven profiles might be a solution there. a profile is basically an additional set of configuration for a project, which may be de-/activated in different ways: by default, depending on the host system, the JDK version, … and they can be de-/activated manually, eg on the command line or in the project properties in your IDE. So for example, you would have a VisualStudio and a GCC profile, activated depending on platform. that means, you could still manually activate GCC and deactivate VisualStudio, if you want to compile with MinGW under windows for example.
http://maven.apache.org/pom.html#Profiles
When you say that the native lib should be pre-compiled and integrated into a dedicated package, I hardly see a reason for adding the nar-plugin info into the POM. Do you think it would make sense to have two different POMs, one for the pure Java user, and one („advanced“ ) for those who also want to compile the native part on their own? For the latter, I think that a mixture of the above mentioned Environment variables, and possibly an XML file could solve most of the problems. As far as I understood, it should be possible to define parts of the POM specifically for the respective target operating system, where the POM is executed.
This is what i am also quite unsure about… as in… i just dont know what would be the best way. The best thing there is probably, looking at how other projects do that. for example, i know that JNA has java and native parts, is maven based, and works perfectly/seamlessly (when you use the lib through maven). if it is not possible to find out how exactly they do it by looking at their code, maybe asking in their forum/mailing-list would help. maybe the NAR plugin guy could help too. there is also some project/database that aims to collect a lot of native access java libs, and has quite some already… don’t remember how to get there though. 
https://jna.dev.java.net/source/browse/jna/trunk/jnalib/
about 1. 2. and 3. … i guess is all true… can’t say much there.
src/test/java in maven (git does not have rules for a structure of java projects), is reserved for unit tests (JUnit), not sure if your tests are unit tests. but yes, all this stuff is usually in a single project in maven. 
i do not know of what nature your experimental stuff is, but if it is possibly to be merged with stock JOCL one day, it should be in branches. that is the big pro of git, that branches are so nice to have, and when you get used to git, you’ll use them for everything.
in the nar plugin source, the guy uses src/it/ for samples. i think… i saw that elsewhere too… i don’t know how much of a standard that is, and what „it“ stands for. i would say src/samples/ is fine too.
generally sounds good
keep it rolling!
btw, i just found out that Neuroph depends on encog and your jocl lib (probably because of encog). as soon as you are ready with maven, i will probably bug them to adopt it too. prosperous times ahead! 
btw2, i love not having to register on this forum!
btw3, it tells me though, that a cow has not 4 legs
I have just pushed JOCL to GitHub - gpu/JOCL: Java bindings for OpenCL . But I’m still not sure whether I’ll switch to GIT for the development. There’s hardly a reason to use such a tool for a codebase of ~20 files, which are only edited by one person, and where the revision graph basically degrades to a ‚linked list‘. Revisions, branches, remotes, index, blobs, trees, staging, collaboration, SSH, … Probably nice, but fairly useless for me - a bunch of „JOCL-backup-[DATE].zip“ archive files would also do the job
Did you have any further ideas concerning the native parts for Maven?
I’ve been reading a little bit about Maven and NAR. There’s a lot of information, and I’m not so sure what I should focus on. Most of the functionalities of Maven are simply not relevant for a project which has no dependencies and does not need plugins, extensions and whatever. The JNA POM says something about “faking out maven” for installing the native libs, but nothing about building the library or how the build process can be configured.
However, as far as I understood it, it should be possible to create a POM which assumes that there are existing native libraries (or possibly the final NAR files?) at some remote repository, and can download the version which is appropriate for the clients system.
This POM could serve as the for the POM which also includes information about the native build process. This extended POM would then inherit all settings, and add some sub-tags in the tag. These sub-tags could or should contain the information which is common for all builds on all platforms. The information which is specific for the target platform would be taken from the “AOL.properties” file, which should contain sensible defaults. (Where this file is located and how it could or should be used or possibly adjusted is not obvious from the documentation.). Finally, there may be settings which are not only specific for the target platform, but maybe even for the target system, namely include paths and linker options. How these settings could be specified sensibly is not yet clear to me.
In general, it’s impressing how complex a simple task can become when someone tries to make it simpler than it inherently is. Once this works, I won’t have to wait very long for the forum threads complaining that it does not really work as it should, and I’ll not be able to help anybody with this -_-
The internet has a scary amount of info about most simple things
In my eyes, the main reason to use git is to make the code more easily available, making it easier to write patches.
If you do not want to accept code from anyone else in any case, i can see that you have problems seeing it as useful.
The thing with maven is similar. For you alone, and others compiling your library, it may not any/much better to use maven. You would use it to make life easier for the users of your library (like jeff from encog, plus i already know 2 other projects that would benefit).
Users of your library will usually be interested in all the native libs you can supply, because in the very end, the software is always intended to run on a clients system, which even for maven based projects, still gets the package with an installer/archive. as your native libs should be tiny, it is more practical to always supply all of them, so the end product can just be copied around without having to separately download the library for the new system.
For what works best for you… i would really recommend asking the JNA guys on IRC/forum/mailing-list, explaining your situation, and asking for advice. they will know best whether their method makes sense for you, and if not, will tell you what else to do, or whom to ask.
Well, of course it’s useful to share the code on a public repository - that’s why I wanted to publish it at http://code.google.com/p/jocl/ (although this is pending for nearly half a year now). I’m just not sure whether GIT is the right choice for such a (structurally) simple project which is maintained by a single developer. At least locally I’ll continue with my existing SVN repository, and depending on the contibutions and feedback on GitHub, I’ll see whether it makes sense to eventually migrate to Git completely…
I also understand the benefits of Maven - once everything is up and running, including the (optional!) native compilation. This is especially the case for a library that should be easy to integrate as a building block into larger projects. But it’s obviously not so easy to get the setup right, regarding the native part. I’m not sure whether this is because Maven is originally not intended for native libraries, or because I’m spoiled by clicking „Compile“ in the IDE. For me, the most promising approrach against the „failed dream of easy compilation on all platforms“ until now was CMake, which did work nicely for JCuda on all platforms. There also seems to be a CMake plugin for Maven, maybe this could also solve some of the issues? At least I’ll try to set up everything to make the precompiled libraries available via Maven as soon as possible. Adding the compilation part using the NAR plugin may be more complicated. Concerning JNA: They don’t use the NAR plugin, and it looks like they don’t target the native compilation issue at all - thus I’m not sure whether they are able and willing to provide help for configuring a native build using the NAR plugin…
EDIT: The deployment of precompiled native libs may not be so difficult … I’ll probably find some … „inspiration“ in this thread
ahh that sounds quite good to me, using cmake to build the native part, if there is even a maven plugin!
about JNA… you mean, the native compilation part is handled totally independent of the java/maven part, and only the end product (native libs) are packed with maven later on? sounds acceptable to me too.
… i won’t read all the thread you linked to now… am tired, sorry.
So, let’s forget about adding the build process to Maven, I think we all agree that this plainly is not possible.
But what can achieved then? I assume that it is not sufficient to declare a JAR (containing the native lib) as a dependency, and unpack it to the „target“ directory when packaging… Should the native lib afterwards be packed into the main JAR, and loaded from there? Or should a single JAR contiaining all native libs be added as a dependency for the main JAR? How should this be deployed, and how should it be used by other Maven users? 
I don’t know what this is all about, but I feel like wasting my time -_-
Moin,
I would prefer the allInOne.jar solution, which encapsulate all JOCL components and the native libraries for all supported platforms in one single jar. At runtime, the current platform can be detected and the corresponding natives extracted and get loaded. On (clean) exit, all loaded natives can be unloaded and deleted.
If you deploy this single jar to the central maven repository (e.g. via sonatype), it will be very easy to use your library. The user must only add a dependency descriptor to his pom.xml. E.g. something like this:
<dependency>
<groupId>org.jocl</groupId>
<artifactId>jocl-binding</artifactId>
<version>0.1.4</version>
</dependency>
All other things will be run automatically; maven will fetch the artifact from the central repository. The jocl user is os independent and can use his software on all supported platforms, without ugly recompiling or repacking.
Regards,
Fancy
Tach
I have already extended the LibUtils class for loading the library from a JAR. It first tries to load the lib as a resource from the JAR, and, if this fails, using the usual System.loadLibrary. I think putting all native libs into a single JAR is the only acceptable solution. For some of the libs (Linux64 and MacOS) I have to rely on contributors, so I’ll have to collect the native libs before each release, but this should be a minor problem.
I’m just not sure if it makes sense to use a build system that can not build. And I’m not familiar with the actual usage and deployment either. In the last days, I have been reading a lot about Maven, and during one websearch the google preview contained an abbreviated text like „Maven … dependency nightmare…“, and I thought: Yes, indeed … But when I clicked the link, the page actually said something like „Maven solves the dependency nightmare“ … well, good to know that …
Considering that the benefit of all this only is that a handful of Maven users may have to make only one mouse click instead of two (before encountering some cryptic error message), the time and work I have to invest for the setup, configuration and hand-tweaking some XML files and the yet-to-come deployment is not really justified. Again: We’re talking about a single, tiny JAR file, right? Others may want to use other build- and deployment tools, and I won’t write their build scripts either. However, I’ll try to dig further into this, maybe one day it turns out to be useful…
bye
Is it right that the POM will not contain any build information, and no dependencies, and that the GIT repository may be deleted, because no source files will be required anyhow? So this all boils down to a tremendously complicated way to upload a JAR somewhere? I simply don’t get it…
Well, I assembled the POM, created GPG keys, created a JIRA account and requested a repository. The latter still pending, we’ll see whether this works. I can not imagine why anybody should want to use such a complicated and poorly documented system, but anyhow…