This is el-job.info, produced by makeinfo version 6.8 from el-job.texi.

INFO-DIR-SECTION Emacs
START-INFO-DIR-ENTRY
* El-Job: (el-job).     Async multicore mapcar.
END-INFO-DIR-ENTRY


File: el-job.info,  Node: Top,  Next: 270,  Up: (dir)

el-job
******

Imagine you have a function you’d like to run on a long list of inputs.
You could run ‘(mapcar #'FN INPUTS)’, but that hangs Emacs until done.

   This library lets you run the same function in many subprocesses (one
per CPU core), each with their own split of the ‘INPUTS’ list, then
merge their outputs and pass it back to the current Emacs.

   In the meantime, current Emacs does not hang at all.

   Best of all, it completes _faster_ than ‘(mapcar #'FN INPUTS)’, owing
to the use of all CPU cores!

   That’s it in a nutshell.  You can look at real-world usage by
searching for "el-job" in these packages:
   • org-mem.el
     (https://raw.githubusercontent.com/meedstrom/org-mem/refs/heads/main/org-mem.el)
   • org-roam-async.el
     (https://raw.githubusercontent.com/meedstrom/org-roam-async/refs/heads/main/org-roam-async.el)

* Menu:

* 2.7.0: 270.
* 2.6.0: 260.
* 2.5.0: 250.
* README for 2.4.8: README for 248.

— The Detailed Node Listing —

2.7.0

* New entry point el-job-parallel-mapcar::

New entry point: ‘el-job-parallel-mapcar’

* Weaknesses::

2.5.0

* Future work::

README for 2.4.8

* Design rationale::
* News 2.4: News 24.
* News 2.3: News 23.
* News 2.1: News 21.
* News 2.0: News 20.
* News 1.1: News 11.
* News 1.0: News 10.
* Limitations::

Design rationale

* Processes stay alive::
* Emacs 30 fast-read-process-output::



File: el-job.info,  Node: 270,  Next: 260,  Prev: Top,  Up: Top

1 2.7.0
*******

   • New entry point: ‘el-job-parallel-mapcar’!  Read more below.
   • Argument ‘:inputs’ are now split differently, for the sake of a
     predictable order of outputs.  It is likely less optimal, but
     hopefully not too much.
   • Argument ‘:require’ now accepts strings in addition to symbols.
     Strings are passed to ‘load’ instead of ‘require’.
   • Value of ‘temporary-file-directory’ no longer added to
     ‘:inject-vars’ for you.
        • Now the only values added are ‘load-path’ and
          ‘native-comp-eln-load-path’.
   • Function ‘el-job-ng-job’ renamed to ‘el-job-ng-get-job’.
        • New class ‘el-job-ng-job’.
   • Delete many aliases to el-job-old.el.

* Menu:

* New entry point el-job-parallel-mapcar::


File: el-job.info,  Node: New entry point el-job-parallel-mapcar,  Up: 270

1.1 New entry point: ‘el-job-parallel-mapcar’
=============================================

Until now, we had to make do with an unwieldy ‘el-job-ng-run’ with ~5
keyword arguments.  And we had to understand asynchronous programming to
use it.

   I’ve long dreamed to be able to brainlessly rewrite a form ‘(mapcar
#'FN INPUTS)’ to something like ‘(multicore-mapcar #'FN INPUTS)’ and
have it Just Work and Just Be Faster.

   Finally, it is done!

   The calling convention is

     (el-job-parallel-mapcar FN INPUTS &optional INJECT-VARS)

   Please see the docstring for what you need to know.

* Menu:

* Weaknesses::


File: el-job.info,  Node: Weaknesses,  Up: New entry point el-job-parallel-mapcar

1.1.1 Weaknesses
----------------

Do not use ‘el-job-parallel-mapcar’ with overly trivial functions.  It
adds some overhead per item, so it actually slows you down if the
function itself only takes microseconds or nanoseconds per invocation.

   Example of a bad use-case:

     (let ((spam (make-list 1000000 "spam")))
       (list (benchmark-elapse (mapcar #'upcase spam))
             (benchmark-elapse (el-job-parallel-mapcar #'upcase spam))))

   Return value:

     (0.482017027 4.793430311)

   Another bad use-case is when plain ‘mapcar’ would’ve been fast
enough.  There is some constant overhead related to spinning up
subprocesses.

   Changing 1000000 from the earlier expression to just 100 reveals the
constant to be ~210ms on my machine:

     (let ((spam (make-list 100 "spam")))
       (list (benchmark-elapse (mapcar #'upcase spam))
             (benchmark-elapse (el-job-parallel-mapcar #'upcase spam))))

   Return value:

     (0.000041974 0.219570936)


File: el-job.info,  Node: 260,  Next: 250,  Prev: 270,  Up: Top

2 2.6.0
*******

   • For ‘el-job-old-launch’, new argument ‘:eval’


File: el-job.info,  Node: 250,  Next: README for 248,  Prev: 260,  Up: Top

3 2.5.0
*******

Released _[2025-10-06 Mon]_, v2.5.0 comes with a variant library
"el-job-ng".

   I find it simpler and easier to reason about.  400 lines of code
instead of 800.

   Some differences:

   • Does _not_ ever keep a process alive
   • Does _not_ merge the subprocesses’ outputs in a bespoke way, just
     uses ‘append’.
   • Does _not_ look up ‘load-history’ to try hard to find an .eln
     variant of your libraries, instead it lets ‘require’ look up
     ‘load-path’ normally.
   • Removed argument ‘:if-busy’, you can manage this yourself with
     ‘el-job-ng-busy-p’ and ‘el-job-ng-kill’.
   • Argument ‘:funcall-per-input’ now takes a function of two (2)
     arguments, not one.
   • Argument ‘:callback’ now takes a function of one (1) argument, not
     two.
   • Added optional argument ‘:eval’.

   The design is always changing in response to my needs, so feel free
to file an issue or email!  It’s instructive to hear about other
people’s needs.

* Menu:

* Future work::


File: el-job.info,  Node: Future work,  Up: 250

3.1 Future work
===============

I may write yet another variant.

   Something that came with experience is that it’s best to make a new
variant library for a narrow use-case, rather than complicate one
library with different code flows.  When it comes to this type of
library, you really want to keep it easy to reason about!

   Ideas as of _[2025-10-06 Mon]_:

File IPC
     Sending input and output by writing and reading files, instead of
     through the pipe connection.

     Theory: performance can sometimes be a lot better, with large
     inputs and outputs.  I would guess it depends a lot on the machine.

     Drawback: if the data sent is sensitive, those files probably
     should be encrypted, and that could negate any performance benefit.

Worker daemons
     Keeping subprocesses alive forever, so they are available at beck
     and call – think worker daemons.

     That’s basically implemented in "el-job-old", and partly why it got
     so hairy, but it could be remade to put this usage
     front-and-center, with a "happy path" UX.

     At this time, I do not have a demanding use-case to experiment
     with, to discover what that happy path should be, how the
     affordances should be.

     If you need this now, I recommend async.el
     (https://github.com/jwiegley/emacs-async).


File: el-job.info,  Node: README for 248,  Prev: 250,  Up: Top

4 README for 2.4.8
******************

* Menu:

* Design rationale::
* News 2.4: News 24.
* News 2.3: News 23.
* News 2.1: News 21.
* News 2.0: News 20.
* News 1.1: News 11.
* News 1.0: News 10.
* Limitations::


File: el-job.info,  Node: Design rationale,  Next: News 24,  Up: README for 248

4.1 Design rationale
====================

I wanted to shorten the round-trip as much as possible, *between the
start of an async task and having the results*.

   For example, say you have some lisp that collects completion
candidates, and you want to run it asynchronously because the lisp you
wrote isn’t always fast enough to avoid the user’s notice, but you’d
still like it to return as soon as possible.

* Menu:

* Processes stay alive::
* Emacs 30 fast-read-process-output::


File: el-job.info,  Node: Processes stay alive,  Next: Emacs 30 fast-read-process-output,  Up: Design rationale

4.1.1 Processes stay alive
--------------------------

In the above example, a user might only delay a fraction of a second
between opening the minibuffer and beginning to type, so there’s scant
room for overhead like spinning up subprocesses that load a bunch of
libraries before getting to work.

   Thus, el-job keeps idle subprocesses for up to 30 seconds after a job
finishes, awaiting more input.

   An aesthetic drawback is cluttering your task manager with many
processes named "emacs".

   Users who tend to run system commands such as ‘pkill emacs’ may find
that the command occasionally "does not work", because it actually
killed an el-job subprocess, instead of the Emacs they see on screen.


File: el-job.info,  Node: Emacs 30 fast-read-process-output,  Prev: Processes stay alive,  Up: Design rationale

4.1.2 Emacs 30 ‘fast-read-process-output’
-----------------------------------------

Some other libraries, like the popular async.el
(https://github.com/jwiegley/emacs-async/), are designed around a custom
process filter.

   Since Emacs 30, it’s a good idea to instead use the _built-in_
process filter when performance is critical, and el-job does so.
Quoting NEWS.30
(https://github.com/emacs-mirror/emacs/blob/master/etc/NEWS.30):

     ** The default process filter was rewritten in native code.
     The round-trip through the Lisp function
     'internal-default-process-filter' is skipped when the process filter is
     the default one.  It is reimplemented in native code, reducing GC churn.
     To undo this change, set 'fast-read-process-output' to nil.


File: el-job.info,  Node: News 24,  Next: News 23,  Prev: Design rationale,  Up: README for 248

4.2 News 2.4
============

   • Jobs must now have ‘:inputs’.  If ‘:inputs’ nil and there was
     nothing queued, ‘el-job-launch’ will no-op and return the symbol
     ‘inputs-were-empty’.


File: el-job.info,  Node: News 23,  Next: News 21,  Prev: News 24,  Up: README for 248

4.3 News 2.3
============

   • Some renames to follow Elisp convention
        • ‘el-job:timestamps’ and friends now ‘el-job-timestamps’.


File: el-job.info,  Node: News 21,  Next: News 20,  Prev: News 23,  Up: README for 248

4.4 News 2.1
============

   • DROP SUPPORT Emacs 28
        • It likely has not been working for a while anyway.  Maybe
          works on the v0.3 branch
          (https://github.com/meedstrom/el-job/tree/v0.3), from 0.3.26+.


File: el-job.info,  Node: News 20,  Next: News 11,  Prev: News 21,  Up: README for 248

4.5 News 2.0
============

   • Jobs must now have ‘:id’ (no more anonymous jobs).
   • Pruned many code paths.


File: el-job.info,  Node: News 11,  Next: News 10,  Prev: News 20,  Up: README for 248

4.6 News 1.1
============

   • Changed internals so that all builds of Emacs can be expected to
     perform similarly well.


File: el-job.info,  Node: News 10,  Next: Limitations,  Prev: News 11,  Up: README for 248

4.7 News 1.0
============

   • No longer keeps processes alive forever.  All jobs are kept alive
     for up to 30 seconds of disuse, then reaped.
   • Pruned many code paths.
   • Many arguments changed, and a few were removed.  Consult the
     docstring of ‘el-job-launch’ again.


File: el-job.info,  Node: Limitations,  Prev: News 10,  Up: README for 248

4.8 Limitations
===============

  1. The return value from the ‘:funcall-per-input’ function must always
     be a list with a fixed length, where the elements are also lists.

     For example, org-mem passes ‘:funcall-per-input
     #'org-mem-parser--parse-file’ to el-job, and if you look in
     org-mem-parser.el
     (https://github.com/meedstrom/org-mem/blob/main/org-mem-parser.el)
     for the defun of ‘org-mem-parser--parse-file’, it always returns a
     list of 5 items:

          (list (if missing-file (list missing-file)) ; List of 0 or 1 item
                (if file-mtime (list file-mtime))     ; List of 0 or 1 item
                found-entries                         ; List of many items
                org-node-parser--found-links          ; List of many items
                (if problem (list problem))))         ; List of 0 or 1 item

     It may look clunky to return sub-lists of only one item, but you
     could consider it a minor expense in exchange for simpler library
     code.

  2. Some data types cannot be exchanged with the children: those whose
     printed form look like ‘#<...>’.  For example, ‘#<buffer
     notes.org>’, ‘#<obarray n=94311>’, ‘#<marker at 3102 in
     README.org>’.

     IIUC, this sort of data only has meaning within the current process
     – so even if you could send it, it would not be usable by the
     recipient anyway.

  3. For now, this library tends to be applicable only to a narrow set
     of use-cases, since you can only pass one ‘:inputs’ list which
     would tend to contain a single kind of thing, e.g.  it could be a
     list of files to visit, to be split between child processes.  In
     many potential use-cases, you’d actually want multiple input lists
     and split them differently, and that’s not supported yet.



Tag Table:
Node: Top185
Node: 2701641
Node: New entry point el-job-parallel-mapcar2525
Node: Weaknesses3244
Node: 2604315
Node: 2504461
Node: Future work5601
Node: README for 2486979
Node: Design rationale7257
Node: Processes stay alive7830
Node: Emacs 30 fast-read-process-output8658
Node: News 249547
Node: News 239855
Node: News 2110097
Node: News 2010422
Node: News 1110633
Node: News 1010852
Node: Limitations11241

End Tag Table


Local Variables:
coding: utf-8
End:
