Source code of JCuda on a public SCM

Btw, @Marco13 , where this java code is?

I mean, is there any version control system for it? (I hope it should be something)

Do you mean the Code of JCuda in general? Probably, because I assume that BinTray also expects some public SCM link, like Maven…

Unfortunately, it’s only publicly available in the archives on the website. Locally, I’m using an SVN. Once I registered https://code.google.com/p/jcuda/ , but never put the code there. I should probably add it to https://github.com/gpu/ . I have scheduled a vacation for beginning of August, maybe I can do it there, but some restructuring and cleanups will be necessary, so I can’t give a precise schedule…

(BTW: I’ll split this into an own thread later)

@Marco13 , that will be great! I will prefer to have it in Github and you count me in for testing and cleaning of the code as well.

My thoughts are that you could start with Github as soon as you will be ready to do it and freshness of the code could be maintained during initial work on the repository :slight_smile:

You could count me in.

@Mysterion I’m still busy and waiting for my vacation to be confirmed (something went wrong in the administration there…)

However, I thought about this, particularly about the right project structure. I’d like to have separate projects for JCuda itself and the JCu* libraries. From what I have read so far, “git submodules” seem to be the way to go here: JCuda could be the main project, and the libraries could be (optional?) submodules. I’ll still have to read a bit more about this, though.

Apart from that, I should also take the chance and put all this into a maven-friendlier format. Currently, the split between “Java” and “JNI” appears at top-level, whereas it should be in “src/main/java” and “src/main/native”, respectively, for each project. Also, I could try to at least cover the Java part with a POM.

If you have any thoughts/hints/tips about this, please let me know.

@Marco13 , sounds good so far to me, probably will be better to start with actual work, and update structure during initial process.

Good luck with your vacation!

@Mysterion Well, I’m not sure. I’d really like to have a sustainable structure that makes it easy to build and maintain the whole thing.

I said…

[QUOTE=Marco13]
Apart from that, I should also take the chance and put all this into a maven-friendlier format. Currently, the split between “Java” and “JNI” appears at top-level, whereas it should be in “src/main/java” and “src/main/native”, respectively, for each project. Also, I could try to at least cover the Java part with a POM.[/QUOTE]
… but now I think that trying to place the native part into a Maven-Structure will not play out so well. The compilation is somewhat tricky, due to the CUDA dependencies. And it works so nicely and smoothly with CMake and its “FindCUDA.cmake” script that I think it would not be worth to try covering this with the “nar-plugin” or the “native” plugin. I found https://bitbucket.org/cowwoc/cmake-maven-project/wiki/Home , and it might be worth a try, but still, the question about the right directory structure is not yet clear.

Currently, I think that a structure like the following might be OK:


Root

    JCudaMain/                A project that does not really contain 
                              anything itself, but refers to all others (as git submodules)
        pom.xml               A root/parent POM that summarizes all others
        CMakeLists.txt        The root CMake file that includes all others

    JCudaCommon/              
        CMake/                Common build files ("FindCUDA.cmake")
        JCudaCommonJNI/       The current "CommonJNI" project
            CMakeLists.txt
            src/

    JCuda/
        JCudaJava/            The Java part of core JCuda 
            pom.xml           The usual...
            src/main/java/             ...Maven...
            src/test/java/                     ...structure
        JCudaDriverJNI/       One of the two native libraries for core JCuda
            CMakeLists.txt
            src/
        JCudaRuntimeJNI/      The other of the two native libraries for core JCuda
            CMakeLists.txt
            src/

    JCublas/
        JCublasJava/          The Java part of JCublas
            pom.xml           The usual...
            src/main/java/             ...Maven...
            src/test/java/                     ...structure
        JCublasJNI/           The native part of JCublas
            CMakeLists.txt
            src/
...

This way, every library could be in one Git project, dividing between the Java- and Native part at the highest level in the Git-Project-directory.

Each library would then probably include “JCudaCommon” as a GIT submodule (since they all build on the native part of this one, and use the common CMake files)

However, I’m not sure about the dependency handling: The Java part of each library refers to the Java part of the core “JCuda” project. Should the core “JCuda” project then be a Git submodule of all libraries? Apart from the pure build/maven dependencies, I hope that Git is able to properly handle submodule-relationships like this…

Maybe @kama or @BinaryLogic have some hints here as well (referring to this thread (in German, sorry))

I think I’ll try to set up the structure as described above, locally, and see how it works out, but would be glad to hear any hints (particularly the “This will not work!” ones ;-))

So I started setting up this structure locally. The native part should nearly be finished. The CMake runs properly. I’ll probably have to adjust some things, so that e.g. the binaries are placed in a directory that can later be picked up by the Maven Assembly, but there are other TODOs first.

The structure in terms of Maven is still unclear. The parent-POM would reside in the “JCudaCommon” project, and I still have to figure out whether/how this will work…

Apart from that, and from what I have read so far, there seem to be some pitfalls with git submodules (they seem to have some shortcomings, and could complicate the whole thing). I also considered to just have one large project in https://github.com/jcuda (at least, I grabbed this one ;-)) that summarizes all libraries. But conceptually, I would prefer to have each library as a separate project. Preferably with the possibility to either

  • compile each project individually OR
  • compile all (core) projects as a whole

So I’m still not sure about the best structure for this. What bugs me at the moment is that the “direction” of the dependencies seems odd:

  • The libraries would be git submodules of the (otherwise empty) “JCudaMain” project
  • The libraries would depend on the “JCudaCommon” project (which in turn would have to be a git submodule of EACH library ?!)

Well, I’ll try to continue with the Java/Maven part today and tomorrow, and see how it goes.

@Marco13 , it looks like you are doing good :slight_smile: I’m unsure if you asking exact questions or it’s just a thoughts shouted in the air?:slight_smile:

:smiley: It’s rather the latter, although implicitly still asking for hints, because I’m horribly unsure about CMake- and Maven „best practices“.

However, the CMake- and POM-part basically work now locally, but there are still some TODOs. I know that it might look ridiculous, but I have uploaded the current state at http://jcuda.org/JCuGit%202015-08-09b%20with%20first%20POMs.zip . „Ridiculous“, because people could just say: „Upload it to GIT and handle it there!“. But I would prefer to do the initial upload with a structure that is at least not complete bogus. The \JCudaMain directory contains a CMake file and a POM that should cover the native- and Java/Maven build part of all (core) libraries, respectively, but it should still be possible to compile each library individually (although I have tested the latter only for one library until now). Just for the case that someone wants to try it out :wink:

Otherwise, I’d start to put this into some GIT repo structure (still locally…). I think that the submodule and dependency handling still has potential for some hassle, but until now it seems to work OK…

*** Edit ***

I’m still not sure whether this submodule stuff will actually work. Each Project needs the „JCudaCommon“ project. So this would have to be a submodule of each project. But then, it would end up as a subdirectory in the directory of EACH library…?! Well, I’ll see…

As expected: It IS a hassle.

(Just in case I did not yet mention it: I am deeply convinced that Git is a really, really bad piece of software. Really bad. It’s a shame that it is hyped so much)

However, I now consider to simply omit this submodule stuff, and upload the projects individually, so that everybody may clone them locally, one by one. It’s horrible from a practical point of view, but … well, I can still provide the source code as a plain ZIP file on the website, so that people can actually use it.

Subtrees seem to be an alternative. In any case, I’ll verify that the current (local) state works as expected (I can’t test it throughoutly at my current PC), and afterwards place it on github (probably, tomorrow or tuesday evening). The “JCudaCommon” project could then be a subtree of every other library, and … maybe, the other libraries all be subtrees of “JCudaMain”. At least, I hope, the basic projects/directory structure should not change significantly…

@Marco13 ,

Btw, probalby you heard about it - https://bitbucket.org/cowwoc/cmake-maven-project/wiki/Home - unsure if it worth while.

Interesting thought about Git :slight_smile: I’m expecting that submodule stuff for Maven works well.

@Mysterion Yes, I already linked to that page in post #6 :wink: It seems rather straightforward, and like it really only calls the command line version of CMake.

Well, Git … if someone is not using it, he is just too stupid to understand the great innovative concepts behind it :twisted: however, one could argue about it endlessly, I’ll try to avoid further bashing here

I don’t see a particular problem for maven. The fact that the „Parent POM“ is currently referred to via a relative path may not be so nice, but since it has to reside in a „sibling“ project of the others, there is probably no other way (except for „inlining“ one project into another … like … some sort of „submodule“). I’ll try to commit a first, initial „Under construction but basically working“ state later today (or early tomorrow).

There will be a bit fidding with the makefiles afterwards. E.g. I’m not sure where to (automatically) place the binaries, so that the Maven tests can be run. (And, by the way: Currently, most of the tests are no automated junit-tests, although for some of them this will just mean adding an @Test instread of using the main()). The Samples, non-Core and „Utils“-projects can probably be added then as well.

Git Submodules would be the wrong way to go IME.

Maven has its dependency resolution and management mechanism, so there is no need for any other dependency mechanism anymore, and Git subtree/submodule is nothing else than a crude way to have projects depend on each other trough sources, thats what you do in languages that do not provide tools like maven.

Instead, you just need to attach the artifact (dll, so etc.) to your build (see the build helper plugin) as artifact, so other projects/modules can depend on it via the standard maven depdency resolution mechanism, also this way you don’t have to fiddle with sources and build the project if you just need the artifact.

Here is an older post that shopws how to configure the build helper plugin to attach so and dlls:
Re: Maven, DLLs and repositories…how ?
Its using jars, but that is up to you.

@maki Thanks for this hint. I did not yet read the entire POM that is listed there, so I still have to figure out what exactly is solved there how. Does it boil down to “Packing a DLL into a JAR” and “Unpacking this DLL from the JAR where it is needed”? At least, from a first skim, since it refers to DLLs, I’m not entirely sure whether it is applicable here. As mentioned above, the projects will be (only listing the relevant parts here)


JCudaCommon/
    JCudaCommonJNI

JCuda/
    JCudaRuntimeJNI
    JCudaDriverJNI
    JCudaJava

JCublas/
    JCublasJNI
    JCublasJava

Every “Java” project depends on JCudaJava. This can easily and obviously be solved with Maven: “JCudaJava” is added as a dependency to “JCublasJava” in the POM.

The tricky part is: Every “JNI” project depends on the “JCudaCommonJNI”. It is a build-time dependency, and not a runtime dependency. The “JCudaCommonJNI” creates a “.lib” file and contains headers that are required in order to build e.g. “JCublasJNI”. So there is no “JCudaCommon.dll” or so created that could be packed into a JAR and referred to as a maven artifact. (One could consider changing this, but then there would still be the build-time dependency to the headers).

(EDIT> And, by the way: The build of the native part is currently done with CMake - whether or not this can be “integrated” into the Maven build, so that Maven is reponsible for copying headers around and starting CMake still has to be figured out <EDIT)

However, I’ll have a closer look at the link that you posted.

@maki From another short look, this looks similar to what @Mysterion is doing with GitHub - MysterionRise/mavenized-jcuda: Mavenized JCuda (But I still have to study it in more detail).

Anyhow: The current state, a first, initial version that ~„should basically be working“, is now at jcuda (Marco Hutter) · GitHub

I’ll probably not „announce“ this publicly in any way yet - I’d like to have it in a more „mature“ state before actually linking to it. Currently, one will have to clone all the projects manually -_- and then run CMake and Maven on the CMakeLists.txt and pom.xml that are contained in „jcuda-main“. This should/could probably be simplified/improved, by adding all libraries as „subtrees“ (?) of „jcuda-main“.

Any feedback will be highly welcome :slight_smile:

But since I slept only ~4 hours last night, that’s all for today

The mavenized-jcuda looks really good from a Maven perspective :slight_smile:

Sorry if I was to brief in my previous post.

In principle it is supposed to work like this:

  1. A module/project creates one or more artifacts (jar, zip, so, dll, you name it)
    1.1 No idea how you create your libraries/shared objects (CMake?), but the build-helper plugin is used to „attach“ those artifacts, ie. you tell it „there is a dll file, it is an artifact“, so it can be used by other projects/modules.
    Don’t worry about the fact that in this example jar files are used.
    [xml]

    org.codehaus.mojo
    build-helper-maven-plugin


    attach-artifacts
    package

    attach-artifact




    ${project.build.directory}/${artifactId}-${version}-${os-platform}-${os-arch}.jar
    jar
    ${os-platform}-${os-arch}





    [/xml]

  2. these Artifacts can be referenced and used by other projects/modules, for DLLs and SOs, you will probably need to copy them to a certain folder/location, this is where the mavenized-jcuda project is using the dependency plugin, in this example all depdencies are copied to target/lib, but you can also specify only one ore more
    [xml]
    org.apache.maven.plugins
    maven-dependency-plugin
    2.9


    package

    copy-dependencies


    ${project.build.directory}/lib
    true



    [/xml]

This way, you only need to build the artifacts once, all others can use it, no need to check out the source and compile it again.

Thanks :slight_smile:

*** Edit ***

[QUOTE=Marco13;121506]

But since I slept only ~4 hours last night, that’s all for today[/QUOTE]

Good job so far, I will take a closer look at what you have today. Still deeply thinking about creating one project with submodules

[QUOTE=maki]
In principle it is supposed to work like this:

  1. A module/project creates one or more artifacts (jar, zip, so, dll, you name it)
    1.1 No idea how you create your libraries/shared objects (CMake?), but the build-helper plugin is used to „attach“ those artifacts, ie. you tell it „there is a dll file, it is an artifact“, so it can be used by other projects/modules.
  2. these Artifacts can be referenced and used by other projects/modules, for DLLs and SOs, you will probably need to copy them to a certain folder/location, this is where the mavenized-jcuda project is using the dependency plugin, in this example all depdencies are copied to target/lib, but you can also specify only one ore more[/QUOTE]

Thanks. I still think that the main issue/difference here is that is is not suffícient to copy a „dependency“ (DLL or SO) to another directory during the maven build, as this depenceny is - until now - completely unrelated to Maven.

When the common JNI project is built, its structure may be


JCudaCommonJNI/
    common.h 

JCudaCommonJNI.binaries/
    common.lib                 // Note: A lib that is used for static linking Not a DLL or SO

When JCublas JNI is built, the structure is roughly


JCublasJNI/
    cublas.h   // Includes "common.h"
    cublas.cpp 

JCublasJNI.binaries/
    jcublas.dll    // Was built from the local sources, AND the common.lib

All this happens without Maven even knowing that something like this is going on :wink: The „Maven CMake Plugin“ (that was already mentioned above) might come into play here, as it might eventually trigger the CMake process, and therefore copy the dependencies (namely, the „common.lib“ and „common.h“) into a directory where the subsequent call to CMake (for the JCublas JNI part) might pick them up. But I guess in order to get this running, there will be a lot of convention configuration required :wink:

*** Edit ***

Regardless of that: This approach could at least be used for copying the DLLs from the native build output to the actual library directories. (Alternatively, the CMakes could be configured to already place them in the right directory - but in any case, the CMake output directory and the Maven input directory have to match…)

This is the problem :wink:
If you let Maven handle the artifacts, you won’t need git submodules/subtrees.

Maven doesn’t have to care about the artifacts themselves and what they are for (DLLs, lib for static linking etc. pp.), as this only concerns CMake.
Maven just needs to know that there are files that are artifacts, so it can make other projects/modules use them.

Not to make Maven deal with the artifacts, but to make CMake actually compile those artifacts, its not a Maven issue :slight_smile:
Maven only needs to know where the files are and what classifiers to assign (classifiers are used to distinguish the native artifacts for different targets, ie. Win x86, Win 64, Linus x86, Linux 64, and so on).

Basically the examples from above will do the trick to make Maven aknowledge the artifacts.
The CMake part is up to you :slight_smile: