Moving to a slightly larger server to see if resources are at issue with rails instance not running forrectly on the T1.micro instance.

Running ansible-playbook install.yml -i staging with the --ask-become-pass flag using my newly created ubuntu user, which has been added to the admin group, with sudo via sudo visudo config.

Had to run sudo apt-get dist-upgrade and sudo apt-get update manually. Actually I didn’t. I had instantiated an Ubuntu 12.10 image, which is outdated. Just re-upped with version 14 point something. ubuntu-14.04-x86_64

Used this little “one line” method to create the self-signed ssh certificate and key.

Hmmm: invalid locale settings: LC_CTYPE=en_US.UTF-8 LANG=en_US.UTF-8 LC_ALL=en_US.UTF-8

Added this to the playbook (top of roles/language/tasks/main.yml):


- name: generate locales
shell: "locale-gen {{ language }}"
sudo: yes


And also ran this in the server terminal:

export LANGUAGE="en_US.UTF-8"
echo 'LANGUAGE="en_US.UTF-8"' | sudo tee --append /etc/default/locale
echo 'LC_ALL="en_US.UTF-8"' | sudo tee --append /etc/default/locale

Provisioning successful!

Run the server. Successful.

But why am I seeing the default Apache page? I guess it runs be default. Stop Apache, Ubuntu. sudo /etc/init.d/apache2 stop.

Now let’s make sure port 433 is opened.

$ sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 127.0.0.1:11211 0.0.0.0:* LISTEN 13470/memcached
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 529/sshd
tcp 0 0 127.0.0.1:5432 0.0.0.0:* LISTEN 860/postgres
tcp 0 0 0.0.0.0:25 0.0.0.0:* LISTEN 633/master
tcp6 0 0 :::22 :::* LISTEN 529/sshd
tcp6 0 0 ::1:5432 :::* LISTEN 860/postgres
tcp6 0 0 :::25 :::* LISTEN 633/master
udp 0 0 127.0.0.1:11211 0.0.0.0:* 13470/memcached

Hmmm. Is nginx running?


sudo /etc/init.d/nginx status
* nginx is not running


Ah.


$ sudo /etc/init.d/nginx status
* nginx is running


Security Warning. Allow in browser. https://staging.usfoodcoop.org/…

There’s the mad cat. But we’re coming along.


$ sudo tail -10 /var/log/nginx/error.log
2015/11/06 01:34:40 [emerg] 2519#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
2015/11/06 01:34:40 [emerg] 2519#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
2015/11/06 01:34:40 [emerg] 2519#0: still could not bind()
2015/11/06 01:35:21 [emerg] 2658#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
2015/11/06 01:35:21 [emerg] 2658#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
2015/11/06 01:35:21 [emerg] 2658#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
2015/11/06 01:35:21 [emerg] 2658#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
2015/11/06 01:35:21 [emerg] 2658#0: bind() to 0.0.0.0:80 failed (98: Address already in use)
2015/11/06 01:35:21 [emerg] 2658#0: still could not bind()
2015/11/06 17:12:58 [error] 29283#0: *3 upstream prematurely closed connection while reading response header from upstream, client: 108.205.62.183, server: staging.usfoodcoop.org, request: "GET / HTTP/1.1", upstream: "http://unix:/home/ubuntu/apps/ofn_america/shared/sock/unicorn.ofn_america.sock:/", host: "staging.usfoodcoop.org"


Same error we were getting on the EC2 instance. But the top output doesn’t look nearly as frightening in terms of maxing out resources. I think the earlier errors might have been from when nginx was trying to start while Apache was bound to port 80 last night.

Let’s get to the bottom of this. Hey! It opened on the second try.

Rolling back, didn’t make any changes to firewall, ipconfig or whatever it is. It’s iptables. Probably the server security needs to be tightened up a bit.

Email configured and working as well, at least as far as Spree test config is concerned! But emails are not coming through. There is a post about this in the community. Add this to ~/.profile; export RAILS_ENV="staging" (~/.profile imports ~/.bash_profile so I just added it there).

Then:


$ source ~/.bash_profile
ubuntu@191:~/apps/ofn_america/current/log$ sudo service monit start
* Starting daemon monitor monit



sudo /etc/init.d/unicorn_ofn_america restart
Couldn't reload, starting 'cd /home/ubuntu/apps/ofn_america/current; bundle exec unicorn -D -c /home/ubuntu/apps/ofn_america/shared/config/unicorn.rb -E staging' instead


Hmmm. Not sure that I’m liking these file permissions:


$ ls -l /home/ubuntu/apps/ofn_america/shared/config/
total 28
-rwxrwxr-x 1 ubuntu admin 664 Nov 6 01:31 application.yml
-rwxrwxr-x 1 ubuntu admin 168 Nov 6 01:31 bugsnag.rb
-rwxrwxr-x 1 ubuntu admin 603 Nov 6 01:31 database.yml
-rwxrwxr-x 1 ubuntu admin 2303 Nov 6 01:31 post-receive
-rwxrwxr-x 1 ubuntu admin 152 Nov 6 01:31 seed.sh
-rw-r--r-- 1 ubuntu admin 1976 Nov 6 01:31 seeds.rb
-rwxr--r-- 1 ubuntu ubuntu 359 Nov 6 01:34 unicorn.rb


I think we might want them to be 664 as per this discussion.

Also trying to figure out how to insert logo. Found my way to a file called /app/controllers/admin/contents_controller.rb which seems to have something to do with:


@preference_sections = [{name: 'Header', preferences: [:logo, :logo_mobile, :logo_mobile_svg]},
{name: 'Home page', preferences: [:home_hero, :home_show_stats]},
{name: 'Producer signup page', preferences: [:producer_signup_pricing_table_html, :producer_signup_case_studies_html, :producer_signup_detail_html]},
{name: 'Hub signup page', preferences: [:hub_signup_pricing_table_html, :hub_signup_case_studies_html, :hub_signup_detail_html]},
{name: 'Group signup page', preferences: [:group_signup_pricing_table_html, :group_signup_case_studies_html, :group_signup_detail_html]},
{name: 'Footer', preferences: [:footer_logo,
:footer_facebook_url, :footer_twitter_url, :footer_instagram_url, :footer_linkedin_url, :footer_googleplus_url, :footer_pinterest_url,
:footer_email, :footer_links_md, :footer_about_url, :footer_tos_url]}]

Spree’s asset pipline is based on Rails’ asset pipeline.

Looks like the basic Spree assets are in /current/app/assets/ and we can override them by placing content in current/vendor/assets, within which there are two dirs, stylesheets, which is empty, and javascript which contains:


angular-backstretch.js angular-google-maps.min.js jquery.adaptivemenu.js textAngular.min.js
angular-flash.min.js angular-scroll.min.js lodash.underscore.js textAngular-sanitize.min.js


I haven’t used Angular, but I think it’s a front-end js library developed maybe by Google. At any rate, there’s no images in here, but there’s tons in current/app/assets/images.

Granted, since we’re using a fork of an old version of Spree, I’m not sure how relevant these docs are.

What method are we using to manage Rails assets?


$ gem list

*** LOCAL GEMS ***

bigdecimal (1.1.0)
bundler (1.10.6)
io-console (0.3)
json (1.5.5)
minitest (2.5.1)
rake (0.9.2.2)
rdoc (3.9.5)



$ bundler
Using rake 10.4.2
Using i18n 0.6.11
Using multi_json 1.11.0
Using activesupport 3.2.21
Using builder 3.0.4
Using activemodel 3.2.21
Using erubis 2.7.0
Using journey 1.0.4
Using rack 1.4.5
Using rack-cache 1.2
Using rack-test 0.6.3
Using hike 1.2.3
Using tilt 1.4.1
Using sprockets 2.2.3
Using actionpack 3.2.21
Using mime-types 1.25.1
Using polyglot 0.3.5
Using treetop 1.4.15
Using mail 2.5.4
Using actionmailer 3.2.21
Using active_link_to 1.0.0
Using active_model_serializers 0.8.1
Using mini_portile 0.6.2
Using nokogiri 1.6.6.2
Using activemerchant 1.48.0
Using arel 3.0.3
Using tzinfo 0.3.44
Using activerecord 3.2.21
Using activeresource 3.2.21
Using acts-as-taggable-on 3.5.0
Using acts_as_list 0.1.4
Using addressable 2.3.3
Using andand 1.3.3
Using rack-ssl 1.3.4
Using json 1.8.3
Using rdoc 3.12.2
Using thor 0.19.1
Using railties 3.2.21
Using angular-rails-templates 0.2.0
Using angularjs-file-upload-rails 1.1.0
Using angularjs-rails 1.2.13
Using ansi 1.4.2
Using atomic 1.1.99
Using awesome_nested_set 2.1.5
Using awesome_print 1.0.2
Using uuidtools 2.1.5
Using aws-sdk 1.11.1
Using bcrypt 3.1.7
Using bcrypt-ruby 3.1.5
Using blockenspiel 0.4.5
Using multi_xml 0.5.5
Using httparty 0.13.1
Using bugsnag 1.5.2
Using cancan 1.6.8
Using xpath 2.0.0
Using capybara 2.2.1
Using timers 1.1.0
Using celluloid 0.15.2
Using chronic 0.10.2
Using chunky_png 1.3.4
Using climate_control 0.0.3
Using cliver 0.3.2
Using cocaine 0.5.7
Using coderay 1.0.9
Using coffee-script-source 1.3.3
Using execjs 2.5.2
Using coffee-script 2.2.0
Using coffee-rails 3.2.2
Using colorize 0.7.7
Using columnize 0.3.6
Using paperclip 3.5.4
Using bundler 1.10.6
Using rails 3.2.21
Using comfortable_mexican_sofa 1.6.24
Using sass 3.3.14
Using compass-core 1.0.3
Using compass-import-once 1.0.5
Using rb-fsevent 0.9.3
Using ffi 1.9.3
Using rb-inotify 0.9.2
Using compass 1.0.3
Using sass-rails 3.2.6
Using compass-rails 2.0.4
Using safe_yaml 0.9.5
Using crack 0.4.1
Using css_parser 1.3.5
Using css_splitter 0.4.1
Using custom_error_message 1.1.1 from git://github.com/jeremydurham/custom-err-msg.git (at master)
Using daemons 1.2.2
Using dalli 2.7.2
Using database_cleaner 0.7.1
Using excon 0.25.3
Using formatador 0.2.4
Using net-ssh 2.6.8
Using net-scp 1.1.2
Using ruby-hmac 0.4.0
Using fog 1.14.0
Using db2fog 0.8.0
Using debugger-linecache 1.2.0
Using debugger-ruby_core_source 1.2.3
Using debugger 1.6.1
Using deface 1.0.0.rc3 from git://github.com/spree/deface.git (at 1110a13)
Using delayed_job 4.0.4
Using delayed_job_active_record 4.0.2
Using orm_adapter 0.5.0
Using warden 1.2.3
Using devise 2.2.8
Using devise-encryptable 0.1.2
Using diff-lcs 1.2.4
Using eventmachine 1.0.3
Using http_parser.rb 0.5.3
Using em-websocket 0.5.0
Using factory_girl 3.3.0
Using factory_girl_rails 3.3.0
Using ffaker 1.15.0
Using figaro 0.7.0
Using foreigner 1.6.1
Using foundation-icons-sass-rails 3.0.0
Using foundation-rails 5.5.0.0
Using foundation_rails_helper 0.4 from git://github.com/willrjmarshall/foundation_rails_helper.git (at rails3)
Using rspec-core 2.14.4
Using rspec-expectations 2.14.0
Using rspec-mocks 2.14.2
Using rspec 2.14.1
Using ruby-progressbar 1.7.1
Using fuubar 1.3.3
Using geocoder 1.1.8
Using gmaps4rails 1.5.6
Using listen 2.2.0
Using lumberjack 1.0.4
Using method_source 0.8.2
Using slop 3.4.5
Using pry 0.9.12.2
Using guard 2.2.4
Using guard-livereload 2.0.1
Using guard-rails 0.4.7
Using guard-rspec 4.0.4
Using zeus 0.15.4
Using guard-zeus 0.0.1
Using haml 4.0.4
Using highline 1.6.18
Using immigrant 0.1.6
Using jquery-rails 2.2.2
Using json_spec 1.1.1
Using kaminari 0.14.1
Using kgio 2.9.3
Using launchy 2.1.2
Using letter_opener 1.0.0
Using libv8 3.16.14.3
Using momentjs-rails 2.5.1
Using money 5.1.1
Using newrelic_rpm 3.12.0.288
Using oj 2.1.2
Using paper_trail 3.0.8
Using parallel 1.4.1
Using parallel_tests 1.3.7
Using xml-simple 1.1.4
Using paypal-sdk-core 0.2.10
Using paypal-sdk-merchant 1.106.1
Using pg 0.13.2
Using websocket-driver 0.3.2
Using poltergeist 1.5.0
Using polyamorous 0.5.0
Using pry-debugger 0.2.2
Using rabl 0.7.2
Using rack-livereload 0.3.15
Using rails-i18n 3.0.1
Using raindrops 0.13.0
Using ransack 0.7.2
Using redcarpet 3.2.3
Using ref 1.0.5
Using representative 1.0.5
Using representative_view 1.2.2
Using roadie 3.0.1
Using roadie-rails 1.0.3
Using rspec-rails 2.14.0
Using rspec-retry 0.4.2
Using select2-rails 3.2.1
Using shoulda-matchers 1.1.0
Using simple_form 2.0.2 from git://github.com/RohanM/simple_form.git (at master)
Using simplecov-html 0.7.1
Using simplecov 0.7.1
Using spinjs-rails 1.3
Using state_machine 1.1.2
Using stringex 1.3.3
Using truncate_html 0.5.5
Using spree_core 1.3.6.beta from git://github.com/openfoodfoundation/spree.git (at 1-3-stable)
Using versioncake 0.4.0
Using spree_api 1.3.6.beta from git://github.com/openfoodfoundation/spree.git (at 1-3-stable)
Using spree_cmd 1.3.6.beta from git://github.com/openfoodfoundation/spree.git (at 1-3-stable)
Using spree_promo 1.3.6.beta from git://github.com/openfoodfoundation/spree.git (at 1-3-stable)
Using spree_sample 1.3.6.beta from git://github.com/openfoodfoundation/spree.git (at 1-3-stable)
Using spree 1.3.6.beta from git://github.com/openfoodfoundation/spree.git (at 1-3-stable)
Using spree_auth_devise 1.3.0 from git://github.com/spree/spree_auth_devise.git (at 1-3-stable)
Using spree_i18n 1.0.0 from git://github.com/spree/spree_i18n.git (at 1-3-stable)
Using spree_paypal_express 2.0.3 from git://github.com/openfoodfoundation/better_spree_paypal_express.git (at 1-3-stable)
Using therubyracer 0.12.0
Using timecop 0.6.2.2
Using turbo-sprockets-rails3 0.3.6
Using turn 0.8.3
Using uglifier 2.7.1
Using unicorn 4.9.0
Using unicorn-rails 1.1.0
Using webmock 1.13.0
Using whenever 0.9.2
Using wicked_pdf 0.11.0
Using wkhtmltopdf-binary 0.9.9.3
Bundle complete! 85 Gemfile dependencies, 207 gems now installed.
Bundled gems are installed into /home/ubuntu/.gem.


Back to the mail issue. There’s a reference to /etc/monit/conf.d/openfoodnetwork, which in our case is called ofn_america


check process openfoodnetwork_dj_worker_0
with pidfile /home/ubuntu/apps/ofn_america/current/tmp/pids/delayed_job.0.pid
start program = "/bin/bash -c 'RAILS_ENV=staging /home/ubuntu/.rbenv/shims/ruby /home/ubuntu/apps/ofn_america/current/script/delayed_job -i 0 start'"
as uid ubuntu and gid ubuntu
with timeout 120 seconds
stop program = "/bin/bash -c 'RAILS_ENV=staging /home/ubuntu/.rbenv/shims/ruby /home/ubuntu/apps/ofn_america/current/script/delayed_job -i 0 stop'"
as uid ubuntu and gid ubuntu
with timeout 120 seconds
if mem is greater than 250.0 MB for 3 cycles then restart


So in case of redeploying I need to update the first line of the ofn_deployment/roles/common/templates/monit.j2 template to match our app name – and consider batch replacing back to openfoodnetwork in the future.

Then you run sudo service monit restart. Monit is similar to debian’s systemd, I think. It runs services, monitors services and files and even alerts you when things go wrong or appear “odd” (based on your config specifications).

Okay so the script referenced by monit config, current/script/delayed_job.sh contains paths specific to a different server:

!/usr/bin/env bash

export HOME="/home/openfoodweb"
export PATH="$HOME/.rbenv/bin:$HOME/.rbenv/shims:$PATH"

$HOME/apps/openfoodweb/current/script/delayed_job $@


So let’s fix that and update with correct paths, locally, and re-upload.

$ rsync -avP delayed_job.sh ofn-939:apps/ofn_america/current/script/

And restart monit again.

Still not exactly working, but I was able to start the server manually in the terminal:


sudo /bin/bash -c 'RAILS_ENV=staging /home/ubuntu/.rbenv/shims/ruby /home/ubuntu/apps/ofn_america/current/script/delayed_job -i 0 start'
[sudo] password for ubuntu:
delayed_job.0: process with pid 32348 started.


So either it’s an issue with monit not having sudo privileges, path(s) being wrong or something like that. I did notice that the file referenced by with pidfile didn’t exist in the specified directory: /home/ubuntu/apps/ofn_america/current/tmp/pids/delayed_job.0.pid.

I need to spend some time with the Monit docs, but in the meantime I’m trying to get monit (ubuntu-14.04-x86_64) to work with the following config files and a RoR app I’m deploying. There’s a delayed job script that monit is supposed to run: /current/script/delayed_job:

#!/usr/bin/env ruby

require File.expand_path(File.join(File.dirname(__FILE__), '..', 'config', 'environment'))
require 'delayed/command'
Delayed::Command.new(ARGV).daemonize

It starts from the CLI as sudo:

sudo /bin/bash -c 'RAILS_ENV=staging /home/ubuntu/.rbenv/shims/ruby /home/ubuntu/apps/myapp/current/script/delayed_job -i 0 start'

The jobs run and /home/ubuntu/apps/myapp/current/tmp/pids/delayed_job.0.pid exists (until I stop the process).

There is the script that’s supposed to tell monit to run the ruby delayed_job script /etc/monit/conf.d/myapp:

check process myapp_dj_worker_0
with pidfile /home/ubuntu/apps/myapp/current/tmp/pids/delayed_job.0.pid
start program = "/bin/bash -c 'RAILS_ENV=staging /home/ubuntu/.rbenv/shims/ruby /home/ubuntu/apps/myapp/current/script/delayed_job.sh -i 0 start'"
as uid ubuntu and gid ubuntu
with timeout 120 seconds
stop program = "/bin/bash -c 'RAILS_ENV=staging /home/ubuntu/.rbenv/shims/ruby /home/ubuntu/apps/myapp/current/script/delayed_job.sh -i 0 stop'"
as uid ubuntu and gid ubuntu
with timeout 120 seconds
if mem is greater than 250.0 MB for 3 cycles then restart

Which is the script that monit is running.

The bash script referenced in the monit script (located in /current/script/) contains:

#!/usr/bin/env bash

export HOME="/home/ubuntu"
export PATH="$HOME/.rbenv/bin:$HOME/.rbenv/shims:$PATH"

$HOME/apps/myapp/current/script/delayed_job $@

Which it appears is supposed to set the $PATH variable and run the ruby delayed_job script. ($@ is a variable that expands to the positional parameters (bash man) so it becomes -i 0 start or -i 0 stop when it calls current/script/delayed_job.

For what it’s worth the path is:

$ echo $PATH
/home/ubuntu/.rbenv/shims:/home/ubuntu/.gem:/home/ubuntu/.rbenv/shims:/home/ubuntu/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games
$ echo $HOME
/home/ubuntu

(The pidfile, /home/ubuntu/apps/myapp/current/tmp/pids/delayed_job.0.pid doesn’t exist either before or after starting (or restarting) monit with sudo service monit start.)

Permissions and ownerships:

-rwxr-xr-x 1 ubuntu ubuntu  175 Nov  6 01:35 delayed_job
-rwxr-xr-x 1 ubuntu ubuntu  175 Nov  6 01:35 delayed_job.sh

I can also, from within the script dir, start the process with CLI:

sudo bash delayed_job.sh -i 0 start

And again, tmp/pids/delayed_job.0.pid exists and delayed jobs (from the DB) are run.

So why won’t monit run it? (I have tried substituting script/delayed_job in the monit config script with script/delayed_job.sh).

Any thoughts where would be a good place to poke about next?

UPDATE:

There is a monit log file (in this case located in /var/log/monit.log, but it’s not terribly revealing:

[EST Nov  9 00:37:32] info     : 'myapp_dj_worker_0' trying to restart
[EST Nov  9 00:37:32] info     : 'myapp_dj_worker_0' start: /bin/bash
[EST Nov  9 00:39:58] error    : 'myapp_dj_worker_0' process is not running

I think the answer lies in this post. You can open a Bash CLI interface within an environment that mirrors the one Monit runs in and try running the commands directly there:

#open the shell with $PATH set
env -i PATH=/bin:/usr/bin:/sbin:/usr/sbin /bin/sh

Within BASH:

$ /bin/bash -c 'RAILS_ENV=staging /home/ubuntu/.rbenv/shims/ruby /home/ubuntu/apps/ofn_america/current/script/delayed_job -i 0 start'
delayed_job.0: process with pid 19825 started.

Seems to be working now. But we’re now getting our old friend the nginx 500 error again:

$ sudo tail -1 /var/logs/nginx/error.log
2015/11/08 09:23:04 [error] 16263#0: *9 connect() to unix:/home/ubuntu/apps/ofn_america/shared/sock/unicorn.ofn_america.sock failed (111: Connection refused) while connecting to upstream, client: 108.205.62.183, server: staging.usfoodcoop.org, request: "GET / HTTP/1.1", upstream: "http://unix:/home/ubuntu/apps/ofn_america/shared/sock/unicorn.ofn_america.sock:/", host: "staging.usfoodcoop.org"

Hmmm.

$ ps aux | grep unicorn
ubuntu   20781  0.0  0.0  10412   960 pts/2    S+   01:34   0:00 grep --color=auto unicorn

Hmmm.

$ uptime
 01:35:05 up 3 days, 56 min,  2 users,  load average: 0.00, 0.05, 0.03

Fuck. We’re not connecting to unicorn again it appears?

$ sudo vim /etc/nginx/sites-available/ofn_america

There’s logs above the current directory in shared/log/unicorn.log.

Hasn’t done anything in two days.

Tried updating the unicorn.rb file:

set path to application

app_dir = File.expand_path("../..", FILE)
shared_dir = "#{app_dir}/shared"
working_directory app_dir

Set unicorn options

worker_processes 2
preload_app true
timeout 30

Set up socket location

listen "#{shared_dir}/sockets/unicorn.sock", :backlog => 64

Logging

stderr_path "#{shared_dir}/log/unicorn.stderr.log"
stdout_path "#{shared_dir}/log/unicorn.stdout.log"

Set master PID location

pid "#{shared_dir}/pids/unicorn.pid"


Unicorn wasn’t running.

`sudo service unicorn_ofn_america start`

Rails is probably supposed to start it. Maybe via monit. Up and running again at the moment any. Might be our resources aren’t enough to handle this app.

After upgrading to newer version of Ruby and/or deployment playbooks, default rake version was older than the gemfile version. Was getting error:

`You have already activated rake 10.1.0, but your Gemfile requires rake 10.4.2. Prepending `bundle exec` to your command may solve this.`

Tried to update it with bundle update rake. Not so. Might not matter as I can run tasks with bundle exec rake.

There are a couple of images in the admin section of the site that are coming up as broken images. Checked the log file at sudo /var/logs/nginx/error.log and lo and behold:

2015/11/24 23:06:06 [error] 8915#0: *1908 upstream prematurely closed connection while reading response
header from upstream, client: 108.205.62.183, server: staging.usfoodcoop.org, request: "GET /logos
/original/missing.png HTTP/1.1", upstream: "http://unix:/home/ubuntu/apps/ofn_america/shared
/sock/unicorn.ofn_america.sock:/logos/original/missing.png", host: "staging.usfoodcoop.org", referrer: 
"https://staging.usfoodcoop.org/"

This may be due to not allowing non secure connection. Let’s look into the nginx configs:

cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.$(date "+%b_%d_%Y_%H.%M.%S")

I think port bindings are defined in /etc/nginx/sites-available/, which contains two files: default and ofn_america.