summaryrefslogtreecommitdiff
path: root/Biz/Bild.hs
AgeCommit message (Collapse)Author
2024-05-20Detect Python `import as` statementsBen Sima
The regex would fail if there was text after the `import X`, so `import X as Y` would fail, and bild wouldn't detect the local import. This adds a simple test that guards against this regex failure.
2024-05-09Turn on more Haskell errorsBen Sima
I actually thought these were turned on, but found that they weren't when I was investigating Haddock stuff. There aren't many violations, so I just turned them on and fixed the violations real quick.
2024-05-09Switch to nixpkgs-23.11, ghc 9.6.3Ben Sima
This brings a bunch of improvements. I got rid of some custom packages, I can now build exllama without using a non-default cuda version. Oh yeah and I get to use GHC 9.6.2 now, a huge upgrade from 9.4. Unfortunately I also updated ormolu and some unrelated formatting changed, but that's life I guess.
2024-04-15Wrap bild log at the terminal-supplied widthBen Sima
Apparently `$COLUMNS` is a POSIX standard, which allows us to set the print width to however wide the user's terminal is. This is a better UI on both wide and narrow terminal layouts: on very narrow layouts, the terminal will properly clear the line instead of doing the wrap-print thing it does when the line overflows, and on wide layouts you can see more of the log message if you're curious. This only works if you export `$COLUMNS` though, because bash only sets the variable in interactive mode, so by default a running program doesn't see it. <https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap08.html>
2024-04-15Cleanup build processes on timeoutBen Sima
On timeout, bild would leave it's child processes still running, which means that a long build would continue to go in the background, and I'd have to manually find and kill it. The fix was simply to use Conduit's `closeStreamingProcessHandle` function, and make sure that Conduit created a group for the processes (by default it doesn't). I tested this by building and running `bild` with a very low timeout (1 minute) and watching `gotop` to make sure it stopped the active build process. I couldn't figure out a simple way to test this automatically.
2024-04-10Explicitly pass Bild.nix to nix builds & expose references to stableBen Sima
This change was motivated by my testing of tabbyAPI. I kept doing like `nix-build -A pkgs.tabbyAPI` and I thought, can't bild just do this? So I wrote a file called TabbyAPI.nix with the following contents:: { bild }: bild.pkgs.tabbyAPI and it worked, I just needed this change to Bild.hs to supply the `bild` argument. The benefit of using bild here is that I can get the logging, concurrency settings, and linking to _/nix etc all by default. Plus, using a standalone nix file like TabbyAPI.nix might be a good way to pin some package in the build system and make sure it continues to build, test, and so on. Also, thie means I don't sprinkle relative paths to the Bild.nix library throughout the repo, which is bad practice anyway. Re: explicitly exposing refernces to stable: This keeps things a bit more tidy and less confusing when working on the nix library.
2024-04-10Fix --cpus and --jobsBen Sima
This patch fixes two problems: 1. Apparently the docopt spec wasn't written correctly, I needed to add a metavar (like N for number) to the spec so that docopt would correctly parse the command line args. Without this, I had just been using the default 6 and 10 that I hardcoded. 2. I never added the --max-jobs and --cores flags to NixBuild, so anytime I built a nix target, i.e. any of my NixOS images, it would use all of the available cores. I think this is part of why I struggled so much to build webkitgtk: I *thought* I was limiting the resources to prevent an OOM crash, but I wasn't. With this fix, I can comfortably limit resources for a build, so I can continue working while building something in the background.
2024-04-10Don't analyze namespaces we can't buildBen Sima
Such a simple change but could potentially save seconds on every bild invocation because I have to go to disk and read the file and so on. I really should have caught this the first time I wrote it.
2024-04-03Hardcode hooksPath in bildBen Sima
This seems to be needed for git test to work, specifically because in a new worktree, git is not configured, and direnv is not loaded to auto-configure these settings. At least that's as far as I can tell what's going on. In either case, best not to rely too much on git anyway.
2024-04-01Make bild exception for git hooksBen Sima
Apparently git hooks don't get called if they have a file extention, so these weren't getting called at all since commit 904de577261e7024373e7a42fd763184764238f9. So this renames them back to the extension-less versions, and adds an exception in bild for files in the core.hooksPath directory. Unfortunately this means Lint.hs will silently ignore these files, but I guess that's okay for now.
2024-04-01Rename .bash files to .shBen Sima
I was already using .sh to refer to shell scripts. When I added .bash, I didn't realize that the linter was set to act on .sh files and not .bash files, so all of these bash scripts escaped the linter. This commit renames them to .sh, removes the .bash extension support from Biz.Namespace, and fixes all the reported shellcheck errors.
2024-04-01Add --jobs and --cpus to bildBen Sima
By default, nix will use the maximum amount of cores available to the machine. On my machine it was maxxing out the CPUs and then actually running out of RAM when compiling JavaScriptCore and literally shutting down my machine. So, I need to be able to control the concurrency and parallelism. The default settings I chose should reserve 4 cores for the user. I also changed --json to --plan because -j makes more sense for --jobs, as its used this way in other tools like make and nix-build.
2023-10-10Add beryllium and connect via VPNBen Sima
I finally got everything setup for the new dev machine, but I ran into a networking problem: I can't tell my home router to expose the ssh port 22 to multiple hosts. I could have made beryllium use a different port, but instead I decided to use tailscale, and this seems to work well. I still don't have hostname routing working, but maybe that's a simple config in tailscale somewhere. Eventually I will get all intra-networking stuff to use a vpn, but for now just using it for beryllium is fine.
2023-10-10Allow for infinite timeout in bildBen Sima
Setting '--time 0' will disable the timeout mechanism altogether. This is necessary because the initial bootstrap build (without access to any cached dependencies) takes *hours* mostly because of cuda and opencv and things like that.
2023-10-10Don't swallow namespace-parsing errorsBen Sima
Previously, if there was a problem with the inputs and bild failed to determine the namespace, 'fromPath' would return 'Nothing' and then 'catMaybes' would drop the error-causing input altogether. In the one time that I had a bad input, this made debugging incredibly difficult. It's always a bad idea to swallow errors silently, so instead lets just kill the program if we have bad inputs.
2023-10-03Add rundeps feature to bildBen Sima
This allows me to specify runtime dependencies, not just system or language deps.
2023-10-03Rename BIZ_ROOT to CODEROOTBen Sima
BIZ_ROOT was too specific. CODEROOT allows for other (non-biz) projects to live in the root of the repo. I didn't want to call it GIT_ROOT because maybe someday I won't want to use git. But I'll never not use code.
2023-09-21Add a 10-minute timeout for all buildsBen Sima
A build should never take more than 10 minutes. If it does, then force the programmer to make stuff faster. This should be a forcing function to either delete unneeded code, or improve the build system.
2023-09-20Prototype MynionBen Sima
This implements a prototype Mynion, my chatbot which will eventually help me write code here. In fact he's already helping me, and works pretty well over xmpp. The prompt is currently not checked in because I'm experimenting with it a lot, and it should probably be a runtime parameter anyways. In the course of writing this I added some helper libraries to get me going, configured black (didn't even know that was possible), and added 'outlines' and its dependencies even though I didn't end up using it. I'll keep outlines around for now, but I'm not sure how useful it really is because afaict its just pre-defining some stop conditions. But it took a while to get it working so I'll just keep it in for now.
2023-08-21Refactor the build system for readabilityBen Sima
Lots of changes here but the code is much improved. The nix code is clearer and structured better. The Haskell code improved in response to the nix changes. I needed to use a qualified path instead of the abspath because the BIZ_ROOT changes based on whether bild runs in nix or runs in the user environment. Rather than passing every argument into Builder.nix, now I just pass the json from bild and deconstruct it in nix. This is obviously a much better design and it only came to be after sleeping on it the other night.
2023-08-17Nixify C buildBen Sima
This is working with libsodium as an example. Its unfortunate that we need the extra ':arg -lsodium' but how else can I get the name of the library for linking? Is that something in the nix attr metadata? Anyway, an optimization for another day.
2023-08-17Nixify rust buildBen Sima
Not getting deps yet but thats okay, I basically need to do a bunch of annoying nix work to get rustPackages into a thing like pythonPackages.
2023-08-16Log and return output to callerBen Sima
The main change here is 'puts' now returns a value, this enables me to collect the value from the conduit source while also doing stuff with it, like printing or logging it as I want. Previously I was running conduit over the source, *and then* kicking off the concurrent processes to wait for the process and collect the output. This would (I think) drain the source before it got to the 'puts' conduit run, and so I wouldn't be able to get the output streamed in real time. It took a lot of refactoring and exploratory programming to get to this relatively-small diff, but now puts works correctly. At least I think it does... it seems to work more reliably from ghci than from the shell. Maybe the shell or TERM is causing nix-store to do some buffering? Maybe I need to use the threaded runtime in GHC? Not sure, but I will look out for this issue and try to identify and fix. Update: yep it was the threaded runtime. I enabled that and now it works in the shell. I squashed that commit into this one.
2023-08-16Run lintBen Sima
Idk why these missed the linter. Probably packages updated in the nixpkgs version bump.
2023-08-07Working nixified python buildBen Sima
This represents quite a few evenings of hacking. It doesn't build all of my Python code, because my Python code is not up to snuff, but it builds the examples and pulls in third party dependencies. Some design points: - I'm using buildPythonApplication in Builder.nix because it was getting way too annoying to wrap the Python script and set PYTHONPATH myself. Easier and more robust to just use the upstream nix builder - Because of this, I had to generate a setup.py. Maybe switch to pyproject.toml in the future, whatever. - Also because of this, Target.wrapper is becoming redundant. I'll just remove it when I get Guile built in nix. - Biz/Bild.nix is getting messy and could use a refactor. - In Builder.nix, I worked around the empty directories bug by just finding and deleting empty directories after unpacking. If its stupid but works it ain't stupid! - I had to touch __init__.py files in all directories before building. Annoying! - `repl` just works, which is awesome - To ensure good Python code, I moved lints and added type checking to the build. So I can't build anything unless it passes those checks. This seems restrictive, but if I want to run some non-passing code, I can still use `repl`, so it's actually not inhibitory.
2023-08-07Add hostname to LocalBen Sima
2023-08-07Set BIZ_ROOT in nix builderBen Sima
2023-08-07Remove unneeded nix-instantiate argumentsBen Sima
2023-08-07Documentation improvementsBen Sima
2023-08-03Add packageSet argumentBen Sima
2023-08-03Move buildPhase to a nix-instantiate argumentBen Sima
2023-08-02Better log messageBen Sima
2023-08-02Factor out nix builder into Haskell.nixBen Sima
This also fixed a bug where every dependency would get pulled into the Haskell target while searching for transitive dependencies.
2023-07-31Implement nix builds for HaskellBen Sima
This is prototype quality. For some reason I think it breaks when doing `build **/*.hs`, which isn't good. But also it's working, and the code feels good. Next I'd like to get Python builds working, as hopefully that will force me to improve the existing code to support a second language.
2023-07-31Capture and return stdout from procBen Sima
2023-07-31Add srcs to TargetBen Sima
Only fully implemented for Haskell at the moment but that is okay.
2023-07-31Prototype nix expression generationBen Sima
There are some limitations to this implementation: 1. Using `runCommand` means this will re-run any time anything changes in the repo. The solution is to use the existing import detection to make a list of source files, and put that into a `stdenv.mkDerivation`, which I'll do next. 2. The `NeatInterpolation` usage is ugly. The templated nix code should be extracted into its own file, such as `Biz/Bild/Builder.nix`. 3. I'm not actually calling it yet. The ideal thing would be to call `nix-instantiate`, get the output drv path, and then call `nix-store --realise` on that. To do that I need to refactor my `proc` function to return stdout to the calling function, and I should probably just make helper functions like `nixInstatiate :: Target -> IO DrvPath` and `nixStoreRealise :: DrvPath -> NixStorePath`, or something like that.
2023-07-28Add NIX_CFLAGS_COMPILE and some depsBen Sima
2023-04-28Parse more meta in scheme codeBen Sima
2023-01-06Use str instead of Text.packBen Sima
2023-01-06Fix guile lib building bugBen Sima
I need to guard on the existence of `out`. This should properly be a `Maybe FilePath`, but to make that change requires a lot of refactoring that I don't want to do right now. Instead, I added it to my todo list for later.
2023-01-05Fix a bug where I couldn't bild from a subdirBen Sima
I need to use the absPath, and the `makeAbsolute` function doesn't actually verify that the path is correct, it just prepends the CWD to the given path.
2023-01-05Also log bild output to a fileBen Sima
2022-10-28Factor out metadata handling into Biz.Bild.MetaBen Sima
The code was becoming repetitive and messy, with functions like 'metaDep' and so on. So that's an indication that they just need to have their own home.
2022-10-26Support building with SDLBen Sima
As a byproduct this also (sorta) generalizes how I pass flags to the C compiler using pkg-config, instead of using the guile-config. Now the 'lib' metadata will be added with 'pkg-config --libs', and the 'sys' metadata will be added with 'pkg-config --cflags'. I'm not *really* sure what the difference is, but if it works it works.
2022-08-09Use wrapper and compiler fields from TargetBen Sima
2022-08-08Remove some TODOsBen Sima
I don't like TODOs in my codebase, I'd rather keep them in org files. Eventually I need a linter that prevents all TODOs from getting into code.
2022-08-05Load asdf before loading the target fileBen Sima
Asdf is required in order to load any third-party packages.
2022-08-05Refactoring to simplify some codeBen Sima
Reduced duplication of out and sysdeps detection. Some other readibility cleanups.
2022-08-05Combine GhcLib and GhcExe into just GhcBen Sima