diff --git a/CHANGELOG.markdown b/CHANGELOG.markdown index ce28452d1..7dc617956 100644 --- a/CHANGELOG.markdown +++ b/CHANGELOG.markdown @@ -5,24 +5,13 @@ This list is not comprehensive, and only lists new features and major changes, but not every minor bugfix. The goatcounter.com service generally runs the latest master. -Changes since rc1 +2021-03-29 v2.0.0 ----------------- -- MFA login was broken 🤦 -- Make sure GoatCounter works even when the `tzdata` package isn't available. -- Revamped integration documentation; what started as a single short page grew - in to a rather long chaotic page so clean that up a bit. - -2021-03-20 v2.0.0-rc1 ---------------------- The version is bumped to 2.0 because this contains a number of incompatible changes: several CLI commands got changed, and it includes some large database migrations – running them is a bit more complex than the standard migrations. -Because this includes so many changes I'll do a Release Candidate first. Please -report issues if you have any problems! Barring unexpected large issues I'll do -a 2.0 release next week or so. - An overview of **incompatible** changes: - There are some rather large changes to the database layout for better @@ -175,6 +164,9 @@ An overview of **incompatible** changes: - Many other minor changes and improvements. +- For changes since RC1 see: + https://github.com/zgoat/goatcounter/compare/v2.0.0-rc1...v2.0.0 + 2020-11-10, v1.4.2 ------------------ diff --git a/README.markdown b/README.markdown index 508f830d1..990e198c9 100644 --- a/README.markdown +++ b/README.markdown @@ -97,14 +97,14 @@ GoatCounter should run on any platform supported by Go, but there are no binaries for them (yet); you'll have to build from source if you want to run it on e.g. FreeBSD or macOS. -Note this README is for the latest master; use the [`release-1.4`][r-1.4] branch -for the 1.4 README. +Note this README is for the latest master; use the [`release-2.0`][r-2.0] branch +for the 2.0 README. Generally speaking only the latest release is supported, although critical fixes (security, data loss, etc.) may get backported to previous releases. [releases]: https://github.com/zgoat/goatcounter/releases -[r-1.4]: https://github.com/zgoat/goatcounter/tree/release-1.4 +[r-2.0]: https://github.com/zgoat/goatcounter/tree/release-2.0 ### Deploy scripts and such @@ -136,9 +136,12 @@ Generally speaking only the latest release is supported, although critical fixes ### Building from source +You need Go 1.16 or newer and a C compiler (for SQLite). If you compile it with +`CGO_ENABLED=0` you don't need a C compiler but can only use PostgreSQL. + Compile from source with: - $ git clone -b release-1.4 https://github.com/zgoat/goatcounter.git + $ git clone -b release-2.0 https://github.com/zgoat/goatcounter.git $ cd goatcounter $ go build -ldflags="-X zgo.at/goatcounter.Version=$(git log -n1 --format='%h_%cI')" ./cmd/goatcounter @@ -156,14 +159,21 @@ To build a fully statically linked binary: -ldflags="-X zgo.at/goatcounter.Version=$(git log -n1 --format='%h_%cI') -extldflags=-static" \ ./cmd/goatcounter -You need Go 1.16 or newer and a C compiler (for SQLite). If you compile it with -`CGO_ENABLED=0 go build` you don't need a C compiler, but can only use PostgreSQL. - It's recommended to use the latest release as in the above command. The master branch should be reasonably stable but no guarantees, and sometimes I don't write detailed release/upgrade notes until the actual release so you may run in to surprises. +You can compile goatcounter without cgo if you're planning to use PostgreSQL and +don't use SQLite: + + $ CGO_ENABLED=0 go build \ + -ldflags="-X zgo.at/goatcounter.Version=$(git log -n1 --format='%h_%cI')" \ + ./cmd/goatcounter + +Functionally it doesn't matter too much, but builds will be a bit easier and +faster as it won't require a C compiler. + ### Running You can start a server with: @@ -174,6 +184,11 @@ The default is to use an SQLite database at `./db/goatcounter.sqlite3`, which will be created if it doesn't exist yet. See the `-db` flag and `goatcounter help db` to customize this. +Both SQLite and PostgreSQL are supported. SQLite should work well for most +smaller sites, but PostgreSQL gives better performance. There are [some +benchmarks over here][bench] to give some indication of what performance to +expect from SQLite and PostgreSQL. + GoatCounter will listen on port `*:80` and `*:443` by default. You don't need to run it as root and can grant the appropriate permissions on Linux with: @@ -190,6 +205,8 @@ This will ask for a password for your new account; you can also add a password on the commandline with `-password`. You must also pass the `-db` flag here if you use something other than the default. +[bench]: https://github.com/zgoat/goatcounter/blob/master/docs/benchmark.markdown + ### Updating You may need to run the database migrations when updating. Use `goatcounter @@ -205,30 +222,17 @@ Use `goatcounter migrate pending` to get a list of pending migrations, or ### PostgreSQL -Both SQLite and PostgreSQL are supported. SQLite should work well for most -smaller sites, but PostgreSQL gives some better performance: - -1. Run with custom `-db` flag: - - $ goatcounter serve -db 'postgresql://dbname=goatcounter' - - Or use a socket: - - $ goatcounter serve \ - -db 'postgresql://host=/run/postgresql dbname=goatcounter sslmode=disable' - - You can also use the `PG*` environment variables: - - $ PGDATABASE=goatcounter DBHOST=/var/run goatcounter serve -db 'postgresql://' +To use PostgreSQL run GoatCounter with a custom `-db` flag; for example: - See `goatcounter help db` and the [pq docs][pq] for more details. + $ goatcounter serve -db 'postgresql://dbname=goatcounter' + $ goatcounter serve -db 'postgresql://host=/run/postgresql dbname=goatcounter sslmode=disable' -2. You can compile goatcounter without cgo if you don't use SQLite: +This follows the format in the `psql` CLI; you can also use the `PG*` +environment variables: - $ CGO_ENABLED=0 go build -ldflags="-X zgo.at/goatcounter.Version=$(git log -n1 --format='%h_%cI')" ./cmd/goatcounter + $ PGDATABASE=goatcounter DBHOST=/run/postgresql goatcounter serve -db 'postgresql://' - Functionally it doesn't matter too much, but builds will be a bit easier and - faster as it won't require a C compiler. +See `goatcounter help db` and the [pq docs][pq] for more details. [pq]: https://pkg.go.dev/github.com/lib/pq#hdr-Connection_String_Parameters diff --git a/cmd/gcbench/gcbench.go b/cmd/gcbench/gcbench.go index 2e1c63b0a..3ba167a0c 100644 --- a/cmd/gcbench/gcbench.go +++ b/cmd/gcbench/gcbench.go @@ -133,12 +133,14 @@ func main() { }) zli.F(err) + first := time.Now().Add(-time.Duration(r.nDays) * 24 * time.Hour) + r.ctx = goatcounter.NewContext(r.db) s := goatcounter.Site{ - Cname: zstring.NewPtr("gcbench.localhost").P, - Plan: goatcounter.PlanBusinessPlus, - Settings: goatcounter.SiteSettings{Public: true}, - // site.FirstHitAt = dates[len(dates)-1] + Cname: zstring.NewPtr("gcbench.localhost").P, + Plan: goatcounter.PlanBusinessPlus, + Settings: goatcounter.SiteSettings{Public: true}, + FirstHitAt: first, } zli.F(s.Insert(r.ctx)) r.ctx = goatcounter.WithSite(r.ctx, &s) diff --git a/deploy/README.markdown b/deploy/README.markdown new file mode 100644 index 000000000..c3f746eaf --- /dev/null +++ b/deploy/README.markdown @@ -0,0 +1,3 @@ +Various ways to deploy GoatCounter; see the subdirectories for details. + +- alpine – Set up an Alpine Linux machine. diff --git a/deploy/alpine/README.markdown b/deploy/alpine/README.markdown new file mode 100644 index 000000000..9b351d394 --- /dev/null +++ b/deploy/alpine/README.markdown @@ -0,0 +1,24 @@ +This sets up a basic GoatCounter installation on Alpine Linux, using SQLite. + +This is also available as a ["StackScript" for Linode][s]; If you don't have a +Linode account yet then [consider using my "referral URL"][r] and I'll get some +cash back from Linode :-) + +It should be fine to run this more than once; and can be used to upgrade to a +newer version. + +You can set the version to use with `GOATCOUNTER_VERSION`; this needs to be a +release on GitHub: + + $ GOATCOUNTER_VERSION=v2.0.0 ./goatcounter-alpine.sh + +You can create additional sites with: + + $ cd /home/goatcounter + $ ./bin/goatcounter db create site -domain example.com -email me@example.com + +Files are stored in `/home/goatcounter`; see `/var/log/goatcounter/current` for +logs; and you can configure the flags in `/etc/conf.d/goatcounter` + +[s]: https://cloud.linode.com/stackscripts/659823 +[r]: https://www.linode.com/?r=7acaf75737436d859e785dd5c9abe1ae99b4387e diff --git a/deploy/StackScript b/deploy/alpine/goatcounter-alpine.sh similarity index 70% rename from deploy/StackScript rename to deploy/alpine/goatcounter-alpine.sh index 7bf75c685..cad7d306d 100755 --- a/deploy/StackScript +++ b/deploy/alpine/goatcounter-alpine.sh @@ -2,38 +2,41 @@ # # # -# +# # -# This is a "StackScript" to deploy GoatCounter on a Linode VPS; it's available -# in Linode as: https://cloud.linode.com/stackscripts/659823 +# This will set up an Alpine Linux machine; environment variables: +# GOATCOUNTER_DOMAIN Domain you'll be hosting GoatCounter on +# GOATCOUNTER_EMAIL Your email address +# GOATCOUNTER_PASSWORD Password to access GoatCounter +# GOATCOUNTER_VERSION GoatCounter version (default: v2.0.0). # -# This script should also work fine outside of Linode; it does assume you're -# using Alpine Linux. +# This is available as a "StackScript" to deploy GoatCounter on a Linode VPS: +# https://cloud.linode.com/stackscripts/659823 # # If you don't have a Linode account yet then consider using my "referral URL" # and I'll get some cash back from Linode :-) # https://www.linode.com/?r=7acaf75737436d859e785dd5c9abe1ae99b4387e # # This script's source at the GoatCounter repo is: -# https://github.com/zgoat/goatcounter/blob/master/deploy/StackScript +# https://github.com/zgoat/goatcounter/blob/master/deploy/alpine # # It should be fine to run this more than once; and can be used to upgrade to a # newer version. # -# You can create additional sites with: +# Files are stored in /home/goatcounter; see /var/log/goatcounter for logs; and +# you can configure the flags in /etc/conf.d/goatcounter. # -# $ cd /home/goatcounter -# $ ./bin/goatcounter db create site -domain example.com -email me@example.com +# You can create additional sites with # -# Files are stored in /home/goatcounter; see /var/log/goatcounter/current for -# logs; and you can configure the flags it starts with in /etc/conf.d/goatcounter. +# $ cd /home/goatcounter +# $ ./bin/goatcounter db create site [..] # # Please report any bugs, problems, or other issues on the GoatCounter issue # tracker, or email me at martin@goatcounter.com # # GoatCounter version to set up. -v=${GOATCOUNTER_VERSION:-v1.4.1} +v=${GOATCOUNTER_VERSION:-"v2.0.0"} set -eu @@ -60,7 +63,7 @@ apk add tzdata grep -q '^goatcounter:' /etc/group || addgroup -S goatcounter grep -q '^goatcounter:' /etc/passwd || adduser -s /sbin/nologin -DS -G goatcounter goatcounter -# Get latest version. +# Get latest version if it doesn't exist yet. mkdir -p /home/goatcounter/bin dst="/home/goatcounter/bin/goatcounter-$v" if [ ! -f "$dst" ]; then @@ -74,7 +77,7 @@ ln -sf "$dst" "/home/goatcounter/bin/goatcounter" # Set up site; this may fail if the site already exists, which is fine. cd /home/goatcounter -./bin/goatcounter db create site -createdb -domain "$GOATCOUNTER_DOMAIN" -email "$GOATCOUNTER_EMAIL" -password "$GOATCOUNTER_PASSWORD" ||: +./bin/goatcounter db create site -createdb -vhost "$GOATCOUNTER_DOMAIN" -user.email "$GOATCOUNTER_EMAIL" -user.password "$GOATCOUNTER_PASSWORD" ||: chown -R goatcounter:goatcounter db # Set up log directory. @@ -97,6 +100,11 @@ pidfile="/run/\${RC_SVCNAME}.pid" output_log="/var/log/\${RC_SVCNAME}/current" error_log="/var/log/\${RC_SVCNAME}/current" +start_pre() { + # Make sure this is correct after updates etc. + setcap 'cap_net_bind_service=+ep cap_sys_chroot=+ep' "\$(readlink "\$command")" +} + depend() { use net use dns @@ -105,15 +113,20 @@ depend() { EOF cat << EOF > /etc/conf.d/goatcounter -# These are the defaults. +# The uncommented values are the defaults. -# Listen on all addressed. +# Listen on all addresses. #GOATCOUNTER_LISTEN=:443 # Location of SQLite3 database file or PostgreSQL connection. GoatCounter is # started from /home/goatcounter. #GOATCOUNTER_DB="sqlite://./db/goatcounter.sqlite3" +# If you use PostgreSQL then URI-type connector is recommended, as OpenRC can't +# deal well with spaces; for example: +#GOATCOUNTER_DB="postgresql:///run/postgresql/goatcounter?sslmode=disable" + + # Other flags to add. See "goatcounter help serve". #GOATCOUNTER_ARGS="-automigrate" EOF diff --git a/docs/benchmark.markdown b/docs/benchmark.markdown new file mode 100644 index 000000000..942c3eaa1 --- /dev/null +++ b/docs/benchmark.markdown @@ -0,0 +1,194 @@ +Here are some benchmarks of the GoatCounter dashboard, just to give an indication +of what to expect with the SQLite and PostgreSQL databases. + +These tests were all performed on Linode (SG) with Alpine Linux 3.13 (28 March +2021), PostgreSQL 13.2, and SQLite 3.34.0 using GoatCounter 2.0.0. Your exact +performance may differ based on VPS provider, location, or even individual VPS +in the same data centre. They're just intended as a rough comparison. I used +Linode because that's what I already use. + +GoatCounter was run with `goatcounter serve`; no extra flags or proxy. To +simulate real-world loading times I ran the benchmarks from my laptop, rather +than the VPS. + +Every configuration is run against SQLite and PostgreSQL with four different +databases: + +- 1M pageviews, evenly spread out over a year and 1,000 paths. +- 1M pageviews, evenly spread out over a year and 50,000 paths. +- 10M pageviews, evenly spread out over a year and 1,000 paths. +- 10M pageviews, evenly spread out over a year and 50,000 paths. + +The performance depends on two factors: the number of pageviews, and the number +of unique paths you have. For example a site with 10M pageviews spread out over +50 paths will be a lot faster than 1M pageviews spread out over 200,000 paths. + +The bencharks were run on the following configurations: + +- Nanode 1GB, 1 core, $5/month +- Linode 2GB, 1 core, $10/month +- Linode 4GB, 2 cores, $20/month +- Linode 8GB, 4 cores, $40/month + +The results aren't completely consistent; when resizing a Linode it may get +moved to a different physical machine, which may have better/worse performance. +It may also be affected by temporary performance spikes and the like. It is the +nature of these kind of cloud things. I didn't make a lot of effort to re-run +them a day later and/or on other locations and such as it's all a lot of work. +This is okay, as the overview is just intended as a rough rule-of-thumb +overview. + + +Results +------- + +The times are the average of 10 requests; after 60 seconds GoatCounter will kill +the query; `>60s` means it timed out. + + +### SQLite + +| | 1G/1c | 2G/1c | 4G/2c | 8G/4c | +| -------------- | ----: | ----: | ----: | ----: | +| 1M/1k week | 1.8s | 1.4s | 1.3s | 0.8s | +| 1M/1k month | 5s | 5.4s | 3.7s | 1.7s | +| 1M/1k quarter | 12.9s | 15.5s | 10.3s | 4s | +| 1M/1k year | >60s | >60s | >60s | 13.6s | +| | | | | | +| 1M/50k week | 1.9s | 1.6s | 1.5s | 0.9s | +| 1M/50k month | 5.4s | 5.5s | 4.2s | 1.9s | +| 1M/50k quarter | 15.8s | 16.4s | 11s | 5s | +| 1M/50k year | >60s | >60s | >60s | 15.3s | +| | | | | | +| 10M/1k week | 19s | 7s | 6.5s | 3.3s | +| 10M/1k month | >60s | >60s | 19.4s | 8.9s | +| 10M/1k quarter | >60s | >60s | >60s | >60s | +| 10M/1k year | >60s | >60s | >60s | >60s | +| | | | | | +| 10M/50k week | >60s | 19.3s | 11.1s | 5.9s | +| 10M/50k month | >60s | >60s | >60s | 16.8s | +| 10M/50k quarter | >60s | >60s | >60s | >60s | +| 10M/50k year | >60s | >60s | >60s | >60s | + + +### PostgreSQL + +| | 1G/1c | 2G/1c | 4G/2c | 8G/4c | +| -------------- | ----: | ----: | ----: | ----: | +| 1M/1k week | 0.2s | 0.1s | 0.6s | 0.1s | +| 1M/1k month | 1.8s | 0.8s | 1.2s | 0.3s | +| 1M/1k quarter | 4.2s | 2.4s | 3.4s | 0.9s | +| 1M/1k year | 5.3s | 3.2s | 3.6s | 1.5s | +| | | | | | +| 1M/50k week | 0.4s | 0.3s | 0.5s | 0.3s | +| 1M/50k month | 1.7s | 0.9s | 1.7s | 0.4s | +| 1M/50k quarter | 4.8s | 2.9s | 3.8s | 1.2s | +| 1M/50k year | 6.7s | 4.5s | 4.9s | 2s | +| | | | | | +| 10M/1k week | 12.4s | 2.4s | 2.1s | 1.9s | +| 10M/1k month | >60s | 9.3s | 2.7s | 1.6s | +| 10M/1k quarter | >60s | 10.9s | 2.4s | 0.9s | +| 10M/1k year | >60s | 10.6s | 2.4s | 0.9s | +| | | | | | +| 10M/50k week | >60s | >60s | 3.9s | 2s | +| 10M/50k month | >60s | >60s | 10.5s | 2.4s | +| 10M/50k quarter | >60s | >60s | 11s | 2s | +| 10M/50k year | >60s | >60s | 11s | 1.9s | + + +The take-away is that PostgreSQL is a lot faster; but I've also spent more time +optimizing PostgreSQL so there may be something to win here. + +Many smaller sites will have much fewer than 1 million pageview and 1,000 paths +though. If you're running GoatCounter on a small website then the cheapest +$5/month should be enough. + +The amount of pageviews it can record isn't benchmarked here, but should be 100/s +even on smaller machines. You'll likely hit in to performance problems on the +dashboard before you hit this. + +You can find the "raw" output from `hey` over here: +https://gist.github.com/arp242/ae9d409e47dfe1021ab0b4ff3e5faba7 + + +Benchmark details +----------------- + +You can use `./cmd/gcbench` to set up a database. The the help on that for some +details. + +I used [hey][hey] to run the actual benchmarks with a simple script: + + #!/bin/sh + # + # https://github.com/zgoat/goatcounter/blob/master/docs/benchmark.markdown + # Requires "hey": https://github.com/rakyll/hey + + url=https://gcbench.arp242.net + start=$(date -d @$(( $(date +%s) - 86400 * 5)) +%Y-%m-%d) # Created DB 5 days ago + run() { + send="$url?period-end=$start&period-start=$(date -d @$(( $(date +%s) - 86400 * $1)) +%Y-%m-%d)" + echo "$send" | tee "gcbench/$1" + ./hey -c 1 -n 10 "$send" | tee -a "gcbench/$1" + } + + mkdir -p gcbench + run 7 + run 30 + run 90 + run 182 + run 365 + + +[hey]: https://github.com/rakyll/hey + + +### PostgreSQL + +PostgreSQL 13.2 was run with the following configuration: + + max_wal_size = 4GB # Default: 1G + min_wal_size = 256MB # Default: 80M + random_page_cost = 0.5 # Better for SSDs. + effective_io_concurrency = 128 # Better for SSDs. + default_statistics_target = 1000 # Sample more in ANALYZE; Default: 100 + +And the following were tweaked based on the Linode instance size: + +Nanode 1GB/1 core $5/month + + max_parallel_workers_per_gather = 1 # Number of cores. + max_parallel_maintenance_workers = 1 # Half the cores. + shared_buffers = 256MB # ~25% of RAM + effective_cache_size = 768MB # ~75% of RAM + maintenance_work_mem = 52MB # ~5% of RAM + work_mem = 21MB # ~2% of RAM + + +Linode 2GB/1 core $10/month + + max_parallel_workers_per_gather = 1 # Number of cores. + max_parallel_maintenance_workers = 1 # Half the cores. + shared_buffers = 512MB # ~25% of RAM + effective_cache_size = 1536MB # ~75% of RAM + maintenance_work_mem = 100MB # ~5% of RAM + work_mem = 40MB # ~2% of RAM + + +Linode 4GB/2 cores $20/month + + max_parallel_workers_per_gather = 2 # Number of cores. + max_parallel_maintenance_workers = 1 # Half the cores. + shared_buffers = 1GB # ~25% of RAM + effective_cache_size = 3GB # ~75% of RAM + maintenance_work_mem = 205MB # ~5% of RAM + work_mem = 82MB # ~2% of RAM + +Linode 8GB/4 cores $40/month + + max_parallel_workers_per_gather = 4 # Number of cores. + max_parallel_maintenance_workers = 2 # Half the cores. + shared_buffers = 2GB # ~25% of RAM + effective_cache_size = 6GB # ~75% of RAM + maintenance_work_mem = 410MB # ~5% of RAM + work_mem = 164MB # ~2% of RAM