Implement Task Distribution Using Java Multithreading
Multi-thread download has been around for a long time, such as flashget and NetAnts.
Protocol Support (the range field specifies the request content range). First, you can read the size of the requested content (that is, the file to be downloaded) and divide several blocks, the block is segmented and distributed to each thread for download. The thread downloads data from the beginning of the segment and ends with the end of the segment. The content downloaded by multiple threads will eventually be written to the same file.
Only useful, work requirements: to assign multiple tasks to multiple Java threads for execution, there will be a list of tasks assigned to the thread policy thinking: known: 1. A list of tasks to be executed, 2. specifies the number of threads to be started. The problem is: what tasks are actually executed by each thread.
Using Java multithreading to implement such a task distribution policy is: the task list is continuously segmented by the number of threads, first ensure the average number of tasks per thread can be allocated, the remaining tasks are appended to the thread in sequence from top to bottom-only in quantity, and the tasks actually executed by each thread are continuous. If there is a small amount of porridge (tasks), the number of threads actually started is equal to the number of tasks. Here, only each thread is able to scan the snow in front of its own door. After the fast completion of the action, it is impossible to see other threads tired again.
The implementation and Demo code is as follows: Implemented by three classes and written in a Java file: taskdistributor is the task distributor, task is the task to be executed, and workthread is the custom working thread. The command mode is used in the Code. If the listener can be configured, it would be even better to use the observer mode to control the UI display. Then, it will be able to realize the dynamic movement of block coloring jumps in the download, this defines the focus of the next step.
The Code contains more detailed annotations, which are easy to understand by reading these annotations and execution results. Main () is the test method
Package COM. alpha. thread; import Java. util. arraylist; import Java. util. list;/*** assign a task list to the thread distributor */public class taskdistributor/*** assigns tasks in the list to each thread and distributes them evenly first, how many elements (list) are appended to the array returned by the preceding thread in sequence) it indicates how many worker threads will be started ** @ Param tasklist * list of tasks to be dispatched * @ Param threadcount * Number of threads * @ return list array, each element contains a list of tasks to be executed by this thread */@ Suppresswarnings ("unchecked") public static list [] distributetasks (list tasklist, int threadcount) 0? Threadcount: remaintaskcount; // array of threads to be started, and list of tasks to be executed by each thread [] tasklistperthread = new list [actualthreadcount]; int taskindex = 0; // redundant tasks after the average allocation, the number of remaining tasks after each append to a thread, re-declare the same variable as remaintaskcount //, otherwise the original value of remaintaskcount will be changed during execution, int remainindces = remaintaskcount; For (INT I = 0; I 0) taskindex + = mintaskcount;> // if there is any remaining one, add it to the thread if (remainindces> 0) // print the task allocation status for (INT I = 0; I return tasklistperthread ;>> Package COM. alpha. thread;/*** the task to be executed, which can be changed to a certain State or called during execution. For example, a task has three states: Ready, running, and completed, by default, the ready state should be further improved. The Listener for task * can be added with state change, which determines the display of the UI */class task /*** execution task */Public void execute () public void setstatus (INT status) public int gettaskid () >
Package COM. alpha. thread; import Java. util. list;/*** custom worker thread, holding the list of tasks assigned to it for execution */class workthread extends thread>
The execution result is as follows. Observe the number and interval of tasks assigned to each Java multithreading. The program ends after all threads have completed the assigned task:
Number of tasks of Thread 0: number of tasks of thread 1 in the 25 interval []: number of tasks of thread 2 in the 25 interval []: number of tasks of thread 3 in the 25 interval: number of worker threads to be started in the interval []: 4. The current thread ID is thread-0. | the task id is: 0. The current thread ID is: thread-3 | task id: 75 the current thread ID is: thread-1 | task id is: 25 the current thread ID is: thread-2 | task id is: 50 The current thread ID is: thread-1 | task id is: 26 the current thread ID is: thread-3 | task id is: 76 the current thread ID is: thread-0 | task id is: 1 The current thread ID is: thread-2 | task id is: 51
The above confession is only a basic effort, and it is really a joke to post it. There are more complex functions.
Java multi-thread download tools make full use of network resources, and Flash get and NetAnts have all been implemented: If a thread downloads the content of the segment to be allocated first, it will help other threads download unfinished data until the task is completed; or the Segment segment of a download thread is too small to be busy, which involves further task allocation. For another example, the above two tools can dynamically increase, decrease, or abort threads. The more complicated the tools are, the more complicated they are. These implementations may define various queues for implementation, for example, incomplete task queues, download task queues, and completed queues.
This article is an English version of an article which is originally in the Chinese language on aliyun.com and is provided for information purposes only. This website makes no representation or warranty of any kind, either expressed or implied, as to the accuracy, completeness ownership or
Java task distribution and collection on a grid
I have an application running on a cluster/grid where I need to run N tasks that do not have to communicate. I only need to collect the result of each task. So I have a Master distributing the tasks to some Slaves (possibly running on different hosts) and combining all the results at the end. As the cluster is controlled by a batch system the configuration of my nodes changes for each run and I get a list of nodes that have been assigned to me for my job. I’m looking for a library (pure Java) to help me with this. I looked at the following: MPJ — doesn’t work for me because of the way that MPJ runs when there are multiple processors available on the same machine. It uses custom class loaders and this gives me problems with a native library that I’m loading (it’s loaded multiple times because the custom class loaders load the class multiple times). Hazelcast — works in principle but it’s not really made for this (I can distribute jobs with a queue and put the results back in another queue but it seems like a bit of an overkill). What I like is that it’s easy to set up the group of nodes (in principle just one needs to be specified and the other nodes can just connect to it). Simon/RMI — I guess I could let each slave register with the master and then let the master distribute jobs to each slave. Or let each slave request a queue where the jobs are queued and a queue where the results should be stored from the master. Cajo — would in principle work but I don’t want to have multicast on the grid network and there seems to be no way around this for Cajo. RabbitMQ — I don’t like to have an extra server running and it’s not pure Java. Same for ZeroMQ. Akka — Seems to be overkill as well. And a lot of configuration to set up the group of nodes. Hadoop — Like Akka seems to be an overkill, especially the configuration to set up the group of nodes. JPPF — Seems to be more suited for setting up a long running cluster of servers and nodes. After my application finishes I need to stop all servers and nodes. Also it seems to rely on Serialization of the Tasks which is not an option for me (see further below) So I would stick with either Hazelcast or Simon. Which one is better suited for this kind of application? Does anyone know another library (not too heavy, not too much configuration). Any other suggestions? Hazelcasts ExecutorService is not an option btw. because I’m using some JNI and so the serialization would be a pain.
2 Answers 2
I finally settled with MPJ. The problem with custom class loaders can simply be circumvented by not using the scripts included in MPJ but instead calling the java program directly with the following parameters:
java class rank mpj-config niodev [additional arguments for the application]
The rank, mpj-config and niodev arguments will be removed by the MPI_Init call. mpj-config is a file listing number of ranks, a switching threshold for the message protocol and a list of hosts with corresponding port number and rank. niodev specifies the communication mechanism (see MPJ-Express documentation for more details). The config file could look like this:
3 131072 a6444@20000@0 a6444@20002@1 a6413@20000@2 It is important to seperate the port numbers on the same host by 2, because MPJ uses the specified port + the next one (so e.g. 20000 and 20001).
Simon and Hazelcast were also good solutions but they were a little bit slower than MPJ. Especially the initialization for both is quite a bit slower.