Inappropriate IOCTL
What the heck does "Errno::ENOTTY: Inappropriate ioctl for device" mean?
I recently encountered a problem that I've had in the past, so I thought I would make a post to remind (mostly myself) how I finally solved it. This helps me avoid a groundhog day situation where I continually make the same mistake and don't remember how I fixed it, since I didn't document it properly.
The Setup
We have a Rails app that deploys using Capistrano. As we merge to our develop
and main
branches, we also have a GitHub actions runner that will perform deployments (assuming the build works and tests pass). So, our setup looks something like this:
Capistrano runs through ssh
, where it connects to GitHub to download the repository. Because this is running in an automated environment, we need to use an SSH key instead of a password to authenticate (we'll see why in just a second).
The file that controls how the deployment works is pretty simple. It's contained in config/deploy/staging.rb
, and looks like this:
set :stage, :staging
set :bundle_without, 'test'
set :rails_env, fetch(:staging)
set :use_sudo, false
set :deploy_to, '/var/www/example.com/stage.api.example.com'
server "stage.api.example.com", user: "deployer-bot", roles: %w{web app db}
set :branch, ENV["REVISION"] || ENV["BRANCH_NAME"] || "develop"
set :ssh_options, {
keys: %w(~/.ssh/id_rsa),
forward_agent: true,
}
Now, when I run bundle exec cap staging deploy
locally, it deploys to the server stage.api.example.com
and places the revision in the /var/www/example.com/stage.api.example.com
directory.
When we add GitHub actions into the mix, though, things get a little dicey. My GitHub actions definition file, located in .github/workflows/staging.yml
, looks like this:
name: Build, Test, and Deploy to Staging
on:
push:
branches:
- develop
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
build-and-test:
runs-on: ubuntu-latest
env:
DB_DATABASE: stage_db
DB_ROOT_USER: root
DB_ROOT_PASSWORD: root
DB_USER: eample_admin
DB_PASSWORD: ${{ secrets.MYSQL_USER_PASSWORD }}
steps:
- name: Set up MySQL
run: |
sudo systemctl start mysql.service
mysql -e 'CREATE DATABASE ${{ env.DB_DATABASE }};' -u${{ env.DB_ROOT_USER }} -p${{ env.DB_ROOT_PASSWORD }}
mysql -e "CREATE USER '${{ env.DB_USER }}'@'localhost' IDENTIFIED BY '${{ env.DB_PASSWORD }}';" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
mysql -e "CREATE DATABASE IF NOT EXISTS ${{ env.DB_DATABASE }};" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
mysql -e "GRANT ALL PRIVILEGES ON ${{ env.DB_DATABASE }}.* to '${{ env.DB_USER }}'@'localhost';" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
mysql -e "FLUSH PRIVILEGES;" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
- name: Install SSH key to Server
uses: shimataro/ssh-key-action@v2
with:
key: ${{ secrets.STAGE_API_DEPLOY_KEY }}
name: github-actions
known_hosts: ${{ secrets.STAGE_API_HOST_KEY }}
config: |
host stage.api.example.com
IdentityFile ~/.ssh/github-actions
IdentitiesOnly yes
ForwardAgent yes
- name: Verify SSH Key
run: cat ~/.ssh/github-actions
- uses: actions/checkout@v2
- name: Set up Ruby Environment
uses: ruby/setup-ruby@v1
with:
ruby-version: 2.6.1
# runs 'bundle install' and caches installed gems automatically
bundler-cache: true
env:
RAILS_ENV: staging
- name: Setup Database
env:
RAILS_ENV: staging
run: bundle exec rake db:setup
- name: Perform Database Migrations
env:
RAILS_ENV: staging
run: bundle exec rake db:migrate
- name: Run specs
env:
RAILS_ENV: staging
run: bundle exec rails spec
deploy-staging:
needs: build-and-test
runs-on: ubuntu-latest
steps:
- name: Install SSH Host Key
uses: shimataro/ssh-key-action@v2
with:
key: ${{ secrets.STAGE_API_DEPLOY_KEY }}
name: github-actions
known_hosts: ${{ secrets.STAGE_API_HOST_KEY }}
config: |
Host stage.api.example.com
IdentityFile ~/.ssh/github-actions
IdentitiesOnly yes
ForwardAgent yes
- uses: actions/checkout@v2
- name: Set up Ruby
uses: ruby/setup-ruby@v1
with:
# # NOTE: This is not needed since we have a .ruby-version file.
# # ruby-version: 2.6.1
bundler-cache: true # runs 'bundle install' and caches installed gems automatically
- name: Install SSH Key
run: |
eval "$(ssh-agent -s)"
ssh-add -D
ssh-add ~/.ssh/github-actions
- name: Check SSH Key Viability
run: |
echo "ls -al" | ssh deployer-bot@stage.api.example.com
- name: Deploy to staging
run: |
eval "$(ssh-agent -s)"
ssh-add -D
ssh-add ~/.ssh/github-actions
bundle exec cap staging deploy
There's a lot going on here, but, essentially, we're setting up an SSH key to the stage.api.example.com server (this becomes important in a second), checking the viability of that SSH key, setting up ruby, and finally deploying using Capistrano. The "Check SSH Key Viability" step is unnecessary now, but it was useful in debugging the underlying issue (up next).
The Problem
Now, even though I was able to deploy to staging locally, when it attempted to deploy on Github actions, the following error was displayed during the Deploy to staging
phase:
Run bundle exec cap staging deploy
#<Thread:0x0000560949aab9c0@/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as deployer-bot@stage.api.example.com: Inappropriate ioctl for device (SSHKit::Runner::ExecuteError)
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/prompt.rb:44:in `noecho': Inappropriate ioctl for device (Errno::ENOTTY)
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/prompt.rb:44:in `ask'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/methods/password.rb:68:in `ask_password'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/methods/password.rb:20:in `authenticate'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:87:in `block in authenticate'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:71:in `each'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:71:in `authenticate'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh.rb:254:in `start'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in `call'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in `with'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:177:in `with_ssh'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:130:in `execute_command'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `tap'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:61:in `test'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/capistrano-passenger-0.2.1/lib/capistrano/tasks/passenger.cap:43:in `block (3 levels) in <top (required)>'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in `run'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deployer-bot@stage.api.example.com: Inappropriate ioctl for device
Caused by:
Errno::ENOTTY: Inappropriate ioctl for device
Tasks: TOP => rvm:hook => passenger:rvm:hook => passenger:test_which_passenger
(See full trace by running task with --trace)
deployer-bot@stage.api.example.com's password:
Error: Process completed with exit code 1.
I spent quite a bit of time trying to figure out what Inappropriate ioctl for device means. I'll save you some time. It means: I require input from a terminal and no terminal is attached to this device. Had I seen the line near the bottom that says deployer-bot@stage.api.example.com's password:
, I probably would've solved this a little faster, but I didn't see that because I was too busy looking through the stack trace. To be fair to me, this is a bit of a red herring, anyway, because the real error should've said something akin to git@github.com:FoamFactory/my-repo's password:
.
The Solution
What was happening was that on the GitHub runner, the SSH command to stage.api.example.com
was working fine. What wasn't working, though, was the step where Capistrano, on stage.api.example.com
, was checking out the codebase from GitHub. This is because the public key for deployer-bot@stage.api.example.com
was not registered with GitHub under my user account, nor was there a deploy key for it for the project in question.
So, it seems as though all we need to do is add a deploy key on GitHub for the appropriate repository with the contents of the file id_rsa.pub
in the .ssh
directory for deployer-bot
on stage.api.example.com
, right? Wrong.
To complicate matters, on stage.api.example.com
, the file ~/.ssh/config
for deployer-bot
looked like this:
Host github.com
HostName github.com
IdentityFile ~/.ssh/github-actions
What this means is that for every host except GitHub, it was using the id_rsa
file located in ~/.ssh
. For GitHub, though, it was using github-actions
. This means that I need to add a deploy key for github-actions.pub
in the same directory. Believe it or not, this still wasn't the complete answer.
Capistrano also needed to be told to use github-actions
. Specifically, this line in the config/deploy/staging.rb
file:
keys: %w(~/.ssh/id_rsa),
needed to be changed to this:
keys: %w(/.ssh/github-actions),
After that, magically, everything seemed to magically work.
Conclusion
I hope documenting this helps you in the future. Quite honestly, I think it's a pretty specific case of misconfiguration on my side (I know this because I also posted this on StackOverflow and didn't get a response in 6 months before I figured it out on my own). That said, documenting it here will at least help me to remember what was going on, and, maybe, just maybe help someone else lost in a sea of SSH redirections.