Rust and Wasm Side-by-Side
As such, I've been reading up the Rust programming language, and, more importantly, how to compile Rust to WebAssembly. After learning from the example module given in the RustWasm book, I decided to convert one of my personal npm libraries from JS -> Rust . The first step in this endeavor was to convert over the existing tests that I had in mocha/chai to Rust's native test format, so that as I added code, I could verify that it was working as I expected it to work . However, the ability to run these tests became an issue when I moved to a WASM target.
One of the big downsides to compiling Rust to WASM is that there isn't a good way to debug code within the browser, and have it link back to the Rust source. The Rust-WASM book notes this in the section on debugging:
Unfortunately, the debugging story for WebAssembly is still immature. On most Unix systems, DWARF is used to encode the information that a debugger needs to provide source-level inspection of a running program. There is an alternative format that encodes similar information on Windows. Currently, there is no equivalent for WebAssembly.
Instead, they recommend using testing (specifically, automated testing) to identify regressions before they make it into the build. This is a great idea, and one I obviously support, given my original writing of tests prior to implementing my library. However, there's one big downside to writing tests in Rust when compiling to WASM, as detailed in the next subsection of the Rust-WASM book:
Note that in order to run the #[test]s without compiler and linker errors, you will need to comment out the crate-type = "cdylib" bits in wasm-game-of-life/Cargo.toml.
Wait... what? In order to run my Rust tests, I need to make changes to my source repository just to get it to build? This is a bit of a non-starter for me, because I want to run on a continuous integration system, where I'm able to build the Rust package, run the tests, build the WASM binary, and then deploy to NPM, all automatically.
In other words, what I want to be able to do is something like this:
# These commands compile (and subsequently test) the native Rust code $ cargo build $ cargo test # This command builds the wasm module $ cargo build --target=wasm32-unknown-unknown
The key here, though, is that I want to be able to do this without making any source file or build-file changes.
I spent quite a bit of time trying to determine what I could do here. One thought that came to mind was that I could have a pre-build script that runs that performs this commenting-out of
#[wasm_bindgen] attributes manually. How this would work is, before building, the script would make copies of everything in the local directory to a sub-directory. It would then switch the
lib, instead of
cdylib and comment out any instances of
#[wasm_bindgen]. Needless to say, I didn't want to go this route - it seemed incredibly fragile and error-prone.
Next, a coworker of mine suggested I separate the native Rust code from the WASM aspect of the code, and make the WASM library utilize the code from the native library, similar to how geotoy handles this. This is a great solution, but, when I set up my library, I couldn't for the life of me figure out how to import an enum or struct into the WASM portion of the library.
Basically, geotoy has a setup where it uses the WASM library to expose a set of functions that serve as the API for the library. These functions then utilize the data structures in
src/lib.rs. However, the data structures themselves aren't exposed. With
cratchit, what I want to expose is the
AccountsChart data structures, as well as the
AccountType enumerations directly, instead of a set of gateway functions that utilize these data structures . I suspect there probably is a way to get it to expose the structs, but I wasn't able to figure it out.
The solution I eventually arrived at doesn't separate the libraries per se. Instead, it makes the
#[wasm_bindgen] attributes conditional on whether or not you're building for a WASM target.
The first thing you need to do is make sure you're using both
lib) in your
[lib] crate-type = ["cdylib", "rlib"]
Next, (this might be specific to me), I had to add
#![feature(custom_attribute)] to the top of my crate attributes (i.e. at the top of
#![feature(custom_attribute)] extern crate cfg_if; extern crate json; extern crate wasm_bindgen; use wasm_bindgen::prelude::*; use cfg_if::cfg_if; use std::collections::HashMap; ...
Next, wherever you previously used
#[wasm_bindgen], make it conditional on the target architecture:
/// Use this instead of #[wasm_bindgen] #[cfg_attr(target_arch = "wasm32", wasm_bindgen)]
Finally, and this might, again, be something that is specific to my use case, you need to pull types out of submodules and into the root crate
src/lib.rs. In other words, in my
src/lib.rs, I had the following:
pub mod accounts; pub mod currency;
And I had two files,
src/accounts.rs, which defined types related to accounts and currencies, respectively.
wasm_bindgen didn't appear to like this, so I moved this code into
src/lib.rs and removed references to the modules in the tests.
What I did to test was not use
wasm-pack initially. Instead, I ran
cargo +nightly build --target=wasm32-unknown-unknown to ensure that no errors were present, and that it created a .wasm file in the appropriate
$ cargo +nightly build --target=wasm32-unknown-unknown ... $ ls -al target/wasm32-unknown-unknown/debug total 9520 drwxr-xr-x@ 13 scottj staff 416 Oct 26 10:15 . drwxr-xr-x 3 scottj staff 96 Oct 26 10:14 .. -rw-r--r-- 1 scottj staff 0 Oct 26 10:14 .cargo-lock drwxr-xr-x 8 scottj staff 256 Oct 26 10:14 .fingerprint drwxr-xr-x 3 scottj staff 96 Oct 26 10:14 build -rw-r--r-- 1 scottj staff 122 Oct 26 10:15 cratchit.d -rwxr-xr-x 2 scottj staff 3549051 Oct 26 10:15 cratchit.wasm drwxr-xr-x 13 scottj staff 416 Oct 26 10:15 deps drwxr-xr-x 2 scottj staff 64 Oct 26 10:14 examples drwxr-xr-x 3 scottj staff 96 Oct 26 10:15 incremental -rw-r--r-- 1 scottj staff 125 Oct 26 10:15 libcratchit.d -rw-r--r-- 2 scottj staff 1311774 Oct 26 10:15 libcratchit.rlib drwxr-xr-x 2 scottj staff 64 Oct 26 10:14 native
And, of course, I made sure the tests ran without any changes:
$ cargo test running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/test_accounts-54475044840749bd running 6 tests test account_type_from_integer ... ok test account_creation ... ok test account_type_from_string ... ok test adding_top_level_accounts_to_accounts_chart ... ok test getting_all_account_ids_in_a_chart ... ok test creating_accounts_chart_from_json ... ok test result: ok. 6 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Running target/debug/deps/test_currency-d2d59a6d3436c103 running 1 test test currency_translation_from_string ... ok test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out Doc-tests cratchit running 0 tests test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out
Now, when you want to build your WASM module, you can run:
And, it will create the packaged WASM library for you in
Bonus: Run tests on Travis and deploy WASM to NPM
This was another area that took a bit of figuring out for me. The problem I ran into with Travis-CI was that it doesn't install
wasm-pack by default. As such, you need to install it manually in a script. However, if you have
cache: cargo enabled, it will hang if
wasm-pack was previously installed. Thus, I added the following to my
.travis.yml file to conditionally install
before_script: | if hash wasm-pack 2>/dev/null; then echo "Wasm-pack already is installed" else curl https://rustwasm.github.io/wasm-pack/installer/init.sh -sSf | sh fi
Now, you can add the following to your
script section build both the WASM module and the native Rust module:
script: - cargo clean - cargo build - cargo test - wasm-pack build --target=nodejs
Note the section that calls
wasm-pack build has an additional argument:
--target=nodejs. If you want to test locally using
npm link, you will need this argument, since it packages the WASM module with a
main parameter in the
And, finally, add the deployment logic:
before_deploy: - cd pkg deploy: provider: npm email: <your email address> on: tags: true skip_cleanup: true api_key: secure: <YOUR_API_KEY>
Note that this deployment logic will only deploy on tagged releases, so you may want to change that if you want different behavior.
One last thing to realize: this currently requires nightly rust. So, you'll need to make sure you're using nightly Rust locally, and you need to make sure that, if you're building on travis, that you allow
beta Rust to fail:
rust: - stable - beta - nightly matrix: allow_failures: - rust: stable - rust: beta
The library in question is Cratchit, which is a library designed to encapsulate accounting data for an accounting application running in electron. It's not important that you have familiarity with this library, but if you want to follow along, feel free. ↩︎
I'm going to gloss over the rest of the adventures regarding me porting to Rust, since it's not super interesting, and involves me fighting a lot with the borrow checker to get things to work. ↩︎
If you're not interested in my musings about the problem and how I came to the solution, feel free to skip to the section on the solution I used, later on in this post. ↩︎
The coworker in question is Johannes Hoff who also is a contributer on the geotoy repo. Interestingly enough, the original author of geotoy, Nick Fitzgerald, was an intern at Mozilla when I first started there. It's amazing to me sometimes how small this world is, given that I haven't seen fitzgen in years, and one of my coworkers happens to have a solution to a problem I have, in collaboration with this other person I worked closely with. Sometimes the wheel of fate astounds me. ↩︎
There probably is a good reason for exposing gateway functions, rather than the underlying data structures themselves, which I will discover at some future point, but for now, my chosen solution seems to be a little cleaner. ↩︎
Make sure that you encrypt your api key. You can do this by retrieving an api key from travis-ci.org and then running:
travis encrypt <api_key> --add deploy.api_key. I waited until I had the configuration otherwise the way I wanted it before doing this. ↩︎