MR2K SOLUTIONS
DAILY RFB (READY FOR BUSINESS) DEVOPS MORNING HEALTH CHECK & HANDOVER RUNBOOK
Version 1.1 • Effective: 09 September 2026 Owner: DevOps / Infrastructure Operations Classification: Internal Company Operations
1. Executive Summary
This document defines the Daily RFB (Ready for Business) process for MR2K SOLUTIONS. The objective is to make the morning operational state visible and testable even when the primary DevOps engineer is unavailable. The checklist starts with server access and resource health, validates Docker and its domain mappings, checks PostgreSQL and MongoDB, tests externally reachable HTTPS endpoints, reviews recent application logs, and ends with a business-flow smoke test and a written status.
2. RFB Status Rules
3. Daily RFB Flow — Morning Sequence
Login to the server using the approved SSH account and verify the correct host.
Record date, operator, server hostname, uptime and current resource state.
Check disk, memory, CPU load, filesystem capacity and Docker daemon status.
List all Docker containers; identify expected containers that are Up versus Exited/Restarting/Unhealthy.
Validate container port mappings and compare them with the approved domain/service inventory.
Check Nginx (or the approved reverse proxy) configuration and service state.
Test every critical domain over HTTPS; verify HTTP status and TLS reachability.
Check PostgreSQL service, connectivity, database/session health, storage/autovacuum indicators, and perform only authorized safe maintenance.
Check MongoDB service and perform a safe connectivity/read-only validation.
Review recent Docker/application logs for fatal errors, restart loops, database connection errors and upstream failures.
Run critical application API/UI smoke tests, including authentication where applicable.
Confirm no unexpected containers, exposed ports, stopped critical services, certificate warnings, disk pressure or unreviewed cleanup candidates exist.
Perform controlled cleanup only where pre-approved, complete the RFB status sheet and communicate GREEN, AMBER or RED to the team.
4. Server Baseline Checks
Run commands read-only unless otherwise stated.
5. Docker Health Check
The Docker inventory is the primary source of truth for what is currently running on the server.
RFB rule: Any critical application container that is Exited, Restarting continuously, or Unhealthy is a minimum AMBER and normally RED if the corresponding business function is unavailable.
6. Domain → Docker Mapping Validation
Maintain an approved service inventory outside this runbook containing: Domain, HTTPS port, reverse-proxy path, upstream host/port, Docker container, image/tag, application owner and criticality. The morning operator checks the live server against that inventory.
7. PostgreSQL Health Check & Maintenance Readiness
Do not modify data during RFB. Use the least-privileged approved database account.
If PostgreSQL is containerized in a future architecture, use the Docker service check and the approved container healthcheck instead of assuming a host service.
7.1 PostgreSQL Read-Only Operational Checks
These checks extend the basic service/connectivity test. They are read-only and should be run before any maintenance. Use the approved least-privileged account where possible; use the postgres OS account only for administrative visibility.
7.2 PostgreSQL Cleanup & Maintenance Rules
RFB is primarily a health check. PostgreSQL data cleanup is NOT a routine morning deletion task. Never DELETE business data, DROP databases/tables, TRUNCATE tables, terminate sessions, or manually remove WAL/data files during RFB unless an approved incident/change procedure explicitly authorizes it.
7.3 PostgreSQL Evidence to Record
Record PostgreSQL service state, version, pg_isready result, database list, any unusual database growth, long-running/blocking query findings, autovacuum concerns, storage usage, and any approved maintenance performed. Never record database passwords or connection secrets.
9. MongoDB Health Check
If the service is named differently (for example mongodb), use the approved unit name in the service inventory.
10. Application / API Full-Flow Smoke Test
The purpose is not to execute every test case. It is to prove the complete business path from internet entry point through reverse proxy, container, application and database.
Open the primary application domain in a browser and confirm the login page loads.
Verify HTTPS certificate is valid and the browser shows no certificate/security warning.
Login with an approved non-production/test account where available. Never store passwords in this document.
Confirm authentication succeeds and the authenticated landing page/dashboard loads.
Open one core business screen and verify data can be read.
Execute one safe read-only or reversible business action that exercises the API path.
Verify the browser/network requests do not show repeated 401/403/404/5xx errors that are unexpected for the test.
Verify a core database-backed read succeeds.
If the application has event/history/notification endpoints, open one representative record and verify it loads.
Logout and confirm the session is invalidated as expected.
Record the smoke-test result and exact endpoint/function tested.
11. HTTP/API Command Examples
Replace placeholders with values from the approved service inventory.
curl -I https://<domain>/curl -sS -o /dev/null -w 'HTTP=%{http_code} TIME=%{time_total}\n' https://<domain>/healthcurl -sS https://<domain>/healthcurl -vk --max-time 15 https://<domain>/api/<safe-read-only-endpoint>Do not use -k for the normal PASS test. It disables certificate verification and should only be used for troubleshooting.
12. Server Cleanup & Capacity Check
Server cleanup must be controlled. First inspect, then obtain/confirm authorization for any change. Do not use broad destructive cleanup commands on production merely to make the server look clean.
Minimum production cleanup authorization rule: if a cleanup command deletes, prunes, vacuums, terminates, rotates, or modifies anything, treat it as a change/maintenance action. Record what was changed, why, who authorized it, and the post-change verification.
12. Logs & Incident Indicators
13. What Non-DevOps Staff Can Safely Check
The following is the minimum visibility workflow for an authorized backup operator.
SSH to the approved server using the documented access method.
Run the read-only server commands in Section 4.
Run docker ps -a and compare container names/status against the service inventory.
Run nginx -t and service-state checks where the operator has sudo permission.
Run the domain curl checks in Section 6.
Run the approved PostgreSQL/MongoDB connectivity checks.
Open the application and perform the documented smoke test.
Capture the output or screenshots of failures and send them to the escalation channel.
Do not restart containers/services, rotate certificates, edit Nginx, alter environment files, delete containers/images, prune Docker, terminate PostgreSQL sessions, run destructive SQL, or perform server/database cleanup unless the incident/change procedure authorizes it.
14. Daily RFB Record
15. Service Inventory Template
16. Escalation Matrix
17. Evidence Collection Standard
Record the exact time of the failed test.
Capture the command output that demonstrates the issue.
Record affected domain, container, endpoint and database if known.
Do not paste passwords, API keys, JWTs, private keys or database credentials into tickets/chats.
Use ticket/incident IDs to correlate repeated failures.
When a service is restored, repeat the original failed test and record the recovery result.
18. Suggested Morning One-Page Quick Run
For experienced operators, the following condensed sequence provides the first-pass readiness view:
hostnamectl --static && uptime && free -h && df -hTsudo systemctl --failedsudo systemctl is-active docker nginx postgresql mongod 2>/dev/null || truedocker ps -a --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'sudo nginx -tsudo ss -lntppg_isreadysudo -u postgres psql -c "SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database WHERE datistemplate=false ORDER BY pg_database_size(datname) DESC;"sudo -u postgres psql -c "SELECT schemaname, relname, n_dead_tup, last_autovacuum FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 10;"docker system dfsudo journalctl --disk-usagemongosh --quiet --eval 'db.adminCommand({ ping: 1 })' 2>/dev/null || truecurl -sS -o /dev/null -w 'HTTPS=%{http_code} TIME=%{time_total}\n' https://<critical-domain>/19. Governance & Maintenance
This runbook must be updated whenever a production domain, Docker container, port mapping, reverse-proxy route, database architecture, critical business flow, database maintenance policy, cleanup/retention policy, or ownership changes. The service inventory is the authoritative list of expected services. The runbook should be reviewed at least quarterly and after major infrastructure changes.
20. Approval
END OF DOCUMENT • MR2K SOLUTIONS
| Document Control | Value |
|---|---|
| Purpose | Provide a repeatable morning full-flow production/server readiness test. |
| Primary outcome | A clear PASS / WARN / FAIL RFB status before business activity starts. |
| Coverage | Linux server, Docker containers, domain mappings, HTTPS/TLS, PostgreSQL, MongoDB, application endpoints and logs. |
| Continuity | A non-DevOps team member must be able to execute the checklist and understand what is running/stopped. |
| Change control | Do not restart, delete, prune, modify configuration, or alter databases during a health check unless authorized. |
| Status | Meaning | Action |
|---|---|---|
| GREEN / PASS | Critical services are healthy and business smoke tests pass. | Business can proceed. |
| AMBER / WARN | Non-critical issue, degraded metric, expected maintenance, or incomplete test. | Record issue, notify owner, continue only if business impact is acceptable. |
| RED / FAIL | Critical application, database, network, TLS, Docker or domain path is unavailable. | Escalate immediately; business readiness is not confirmed. |
| Check | Command | Expected Result |
|---|---|---|
| Identity | hostnamectl --static; whoami | Correct server and approved operator. |
| Uptime | uptime | Server is up; load is reasonable for workload. |
| OS | cat /etc/os-release | Expected Ubuntu/Linux release. |
| CPU / memory | free -h; nproc | No unexpected memory exhaustion or CPU pressure. |
| Disk | df -hT | Critical filesystems have safe free space; investigate >80–85%. |
| Inodes | df -ih | No filesystem inode exhaustion. |
| Failed services | systemctl --failed | No unexpected failed critical services. |
| Docker | systemctl is-active docker; docker info | Docker is active and responding. |
| Purpose | Command | What to record |
|---|---|---|
| All containers | docker ps -a --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}' | Name, image/tag, status, ports. |
| Running only | docker ps | Expected production services are Up. |
| Restart loops | docker ps -a --filter status=restarting | Normally empty. |
| Exited containers | docker ps -a --filter status=exited | Investigate any critical service. |
| Health status | docker ps --format '{{.Names}}\t{{.Status}}' | Healthy where healthchecks exist. |
| Docker networks | docker network ls | Expected networks exist. |
| Images | docker images --format 'table {{.Repository}}\t{{.Tag}}\t{{.CreatedSince}}' | Confirm approved image/tag is deployed. |
| Validation | Command / Method | Expected |
|---|---|---|
| Listening ports | sudo ss -lntp | Only approved public/service ports are listening. |
| Docker port map | docker ps --format 'table {{.Names}}\t{{.Ports}}' | Expected host-to-container mappings. |
| Nginx state | sudo systemctl is-active nginx | active |
| Nginx config | sudo nginx -t | syntax is ok; test is successful |
| Nginx sites | ls -la /etc/nginx/sites-enabled/ | Expected sites are enabled. |
| DNS resolution | getent hosts <domain> | Domain resolves to expected server/load balancer. |
| HTTPS | curl -sS -o /dev/null -w '%{http_code} %{remote_ip}\n' https://<domain>/ | Expected HTTP status, typically 200/204/3xx depending on endpoint. |
| Check | Command / Method | Expected |
|---|---|---|
| Service | sudo systemctl is-active postgresql | active |
| Version | psql --version | Expected major version. |
| Connection | pg_isready | accepting connections |
| Sessions | sudo -u postgres psql -c '\l' | Database server responds; expected databases visible. |
| Read-only query | psql '<approved connection string>' -c 'SELECT now();' | Returns current DB time. |
| Errors | sudo journalctl -u postgresql --since 'today' --no-pager | tail -n 100 | No unexplained fatal/startup/storage errors. |
| Check | Command / Method | Expected / Action |
|---|---|---|
| Database list | sudo -u postgres psql -c '\l' | Expected production databases are present; investigate unexpected databases. |
| Database sizes | sudo -u postgres psql -c "SELECT datname, pg_size_pretty(pg_database_size(datname)) FROM pg_database WHERE datistemplate=false ORDER BY pg_database_size(datname) DESC;" | No unexpected rapid growth; investigate databases consuming unusual space. |
| Active sessions | sudo -u postgres psql -c "SELECT datname, usename, state, count(*) FROM pg_stat_activity GROUP BY datname, usename, state ORDER BY count(*) DESC;" | No unexpected session surge or excessive idle/active connections. |
| Long-running queries | sudo -u postgres psql -c "SELECT pid, datname, usename, now()-query_start AS duration, state, left(query,120) FROM pg_stat_activity WHERE query_start IS NOT NULL AND now()-query_start > interval '10 minutes' AND state <> 'idle' ORDER BY query_start;" | Normally empty. Investigate before terminating anything. |
| Dead tuples / autovacuum | sudo -u postgres psql -c "SELECT schemaname, relname, n_live_tup, n_dead_tup, last_autovacuum, last_autoanalyze FROM pg_stat_user_tables ORDER BY n_dead_tup DESC LIMIT 20;" | High dead tuples or stale autovacuum/analyze dates require review. |
| Blocking / waits | Review pg_stat_activity and pg_locks for blocked sessions. | No unexplained blocking of critical application queries. |
| Replication (if applicable) | sudo -u postgres psql -c 'SELECT * FROM pg_stat_replication;' | Expected replicas connected and healthy; N/A for standalone PostgreSQL. |
| PostgreSQL storage | sudo du -sh /var/lib/postgresql /var/log/postgresql 2>/dev/null | Storage should remain within approved capacity; investigate rapid growth. |
| Maintenance Area | Safe RFB Action | Do Not Do During Normal RFB |
|---|---|---|
| Dead tuples / vacuum | Review n_dead_tup and autovacuum timestamps; record candidates for maintenance. | Do not manually VACUUM FULL or change autovacuum settings without authorization. |
| Long-running queries | Identify and report queries running beyond the approved threshold. | Do not pg_terminate_backend() unless incident procedure authorizes it. |
| Database growth | Review database/table size and compare with baseline. | Do not delete rows or archive data without the application/data-owner procedure. |
| PostgreSQL logs | Check size and recent errors under /var/log/postgresql or journal. | Do not rm active PostgreSQL logs manually. |
| Backups | Verify the approved backup job/status if available. | Do not delete backup files as part of RFB. |
| Check | Command / Method | Expected |
|---|---|---|
| Service | sudo systemctl is-active mongod | active, if host-installed. |
| Version | mongosh --version | Expected client version. |
| Ping | mongosh --quiet --eval 'db.adminCommand({ ping: 1 })' | ok: 1 |
| Server status | mongosh --quiet --eval 'db.serverStatus().ok' | 1 |
| Errors | sudo journalctl -u mongod --since 'today' --no-pager | tail -n 100 | No unexplained fatal/storage/startup errors. |
| Area | Inspect / Read-Only Command | Expected | Cleanup Rule |
|---|---|---|---|
| Disk usage | df -hT | Critical filesystems preferably below 80–85%. | Investigate largest consumers before deleting anything. |
| Inodes | df -ih | No inode exhaustion. | Locate high-file-count directories before cleanup. |
| Top filesystem usage | sudo du -xhd1 / 2>/dev/null | sort -h | Understand major space consumers. | Delete only approved application/temp data. |
| Docker disk | docker system df | Usage understood; no unexpected accumulation. | Remove stopped containers/images only when approved and after confirming rollback needs. |
| Stopped containers | docker ps -a --filter status=exited | Normally empty for production. | Remove only verified obsolete containers; confirm no required mounts/configuration. |
| Unused images | docker images | Approved rollback images retained. | Do not run docker image prune -a routinely; remove only explicitly approved images. |
| Nginx logs | sudo du -sh /var/log/nginx; sudo tail -n 100 /var/log/nginx/error.log | No uncontrolled log growth; no repeated critical errors. | Use approved logrotate/retention process, not manual deletion of active logs. |
| System journal | sudo journalctl --disk-usage | Journal storage within approved limit. | Use approved journal retention policy; do not vacuum blindly. |
| APT cache | sudo du -sh /var/cache/apt 2>/dev/null | No abnormal cache growth. | If approved, use package-manager cleanup; do not remove packages to free space. |
| Temporary files | sudo du -sh /tmp /var/tmp 2>/dev/null | No abnormal growth. | Delete only known-safe stale temporary files; preserve application/runtime files. |
| Indicator | Command | Escalation Trigger |
|---|---|---|
| Container logs | docker logs --tail 200 <container> | Repeated ERROR/FATAL, startup failure, DB connection failure. |
| Container events | docker events --since 30m --until 0s | Unexpected restart/stop activity. |
| Nginx errors | sudo tail -n 200 /var/log/nginx/error.log | Repeated upstream timeout/refused/5xx. |
| System journal | sudo journalctl --since 'today' -p err..alert --no-pager | New critical system errors. |
| Disk pressure | df -hT | Critical filesystem near capacity. |
| Memory pressure | free -h; sudo dmesg -T | grep -i -E 'oom|out of memory' | tail | OOM events or severe memory pressure. |
| Field | Entry |
|---|---|
| Date / Time | ____________________________ |
| Operator | ____________________________ |
| Server hostname | ____________________________ |
| Overall status | GREEN / AMBER / RED |
| Docker status | PASS / WARN / FAIL |
| Domain/HTTPS status | PASS / WARN / FAIL |
| PostgreSQL | PASS / WARN / FAIL / N/A |
| MongoDB | PASS / WARN / FAIL / N/A |
| Application smoke test | PASS / WARN / FAIL |
| Critical issue / ticket | ____________________________ |
| Escalated to | ____________________________ |
| Business notified | YES / NO |
| Operator sign-off | ____________________________ |
| Service | Domain | Container | Host Port | Container Port | DB | Criticality | Owner |
|---|---|---|---|---|---|---|---|
| Example – Application A | app.example.com | app_a | 443/proxy | 8080 | PostgreSQL | Critical | Team/Owner |
| Example – Auth | auth.example.com | auth_service | 443/proxy | 8072 | PostgreSQL/MongoDB | Critical | Team/Owner |
| Add actual services from server inventory |
| Severity | Example | Immediate Action | Owner |
|---|---|---|---|
| P1 / RED | Primary application unavailable; database unavailable; HTTPS broken for critical domain. | Stop RFB as PASS; escalate immediately; preserve evidence. | DevOps + application owner + management |
| P2 / AMBER | One non-critical service unhealthy; elevated errors; degraded endpoint. | Record, notify owner, assess business impact. | DevOps / service owner |
| P3 / INFO | Expected warning, scheduled maintenance, cosmetic/log noise. | Record for follow-up. | Service owner |
| Role | Name | Signature | Date |
|---|---|---|---|
| DevOps / Infrastructure Owner | ________________ | ________________ | ____________ |
| Engineering Manager | ________________ | ________________ | ____________ |
| Operations / Business Owner | ________________ | ________________ | ____________ |