# SAP # SAP Production Incident – Troubleshooting SAP Router, APP Dispatcher, and HANA Dependency Failure # ## Introduction Recently I handled an SAP Production incident where users were suddenly unable to access critical SAP production services including SAP GUI At first glance the issue looked like a network or firewall problem because users received a connection refused error when trying to connect to the SAP environment. However, after tracing the entire SAP landscape step-by-step, the actual issue turned out to be a startup dependency problem involving: - SAP Router - SAP Application Dispatcher - SAP HANA Database - Web Dispatcher backend connectivity This article shares the troubleshooting process from infrastructure level until application recovery. > Note: Hostnames, IP addresses, SID names, and environment details have been anonymized for confidentiality.

---

# Environment Overview The SAP production landscape consisted of several separate virtual machines:

1. SAP Router VM

2. SAP Application VM

3. SAP Web Dispatcher VM

4. SAP Database VM

Architecture flow:

Client

↓

SAP Router

↓

SAP APP Dispatcher

↓

SAP HANA Database

---

# Initial Symptoms Users reported:

SAP GUI connection failed

web application inaccessible

SAP GUI error:

partner '<APP-IP>:3230' not reached

Connection refused

At this stage there were several possible causes: - Firewall issue - NAT issue - SAP Router down - Dispatcher down - Database issue - SAP service startup failure

---

# Step 1 – Validate Network Connectivity Initial external validation was performed using:

nmap -Pn -sT -sV \

-p 3299,3230,3330,50013,50014 <PUBLIC-IP>

Result:

3299 closed

3230 closed

50013 open

50014 open

Interesting finding:

SAPControl ports were alive

but SAP application ports were closed.

This usually means:

sapstartsrv running

but SAP application services not operational

---

# Step 2 – SAP Router Investigation SAP Router port:

3299

was confirmed not listening. Further inspection on the SAP Router VM revealed that SAP Router was started manually using:

./saprouter -r -s 3299 ...

It was not configured as a proper persistent service. Additional discovery: The startup script previously used an incorrect parameter:

-S 3299

instead of:

-s 3299

After correcting and starting the service manually:

./saprouter -r -s 3299 ...

Result:

3299 LISTEN

SAP Router connectivity was restored successfully.

---

# Step 3 – SAP APP Instance Investigation Next, SAP instance status was checked:

sapcontrol -nr 30 -function GetProcessList

sapcontrol -nr 31 -function GetProcessList

Initial status: ## Instance 31

msg_server GRAY

enq_server GRAY

## Instance 30

disp+work GRAY

gateway stopped

Port validation:

3230 CLOSED

This explained the SAP GUI connection refusal.

---

# Step 4 – Recover SAP Central Services Instance 31 was started first:

sapcontrol -nr 31 -function StartSystem

Result:

msg_server GREEN

enq_server GREEN

However, instance 30 still failed to become healthy.

---

# Step 5 – Dispatcher Failure Analysis Dispatcher logs were analyzed:

cd /usr/sap/<SID>/D30/work

Important logs:

dev_disp

sapstart.log

Critical error found:

DpWpCheck: no more work processes

Multiple work processes terminated unexpectedly. At this point the investigation shifted toward database dependency.

---

# Step 6 – HANA Database Investigation HANA status check:

HDB info

Result:

HANA services not fully running

Important HANA services such as: - nameserver - indexserver - xsengine were unavailable. This explained why:

SAP APP work processes failed during startup.

The SAP dispatcher depended on HANA connectivity.

---

# Step 7 – Start HANA Database Database services were started manually:

HDB start

After startup:

HANA processes became active

Ports became available again.

---

# Step 8 – Recover SAP APP Instance After HANA became healthy:

sapcontrol -nr 30 -function StartSystem

Result:

disp+work GREEN

gwrd GREEN

icman GREEN

Port validation:

3230 LISTEN

SAP GUI connectivity was restored.

---

# Step 9 – Verify Web Dispatcher Web Dispatcher status:

sapcontrol -nr 00 -function GetProcessList

Initially:

YELLOW

Later:

GREEN

Backend connectivity recovered automatically after APP and HANA services became available. Web application login page became accessible again.

---

# Root Cause Analysis The incident was ultimately caused by:

Several SAP services did not auto-start properly

after system startup/reboot.

Affected components: - SAP Router service - SAP HANA Database services - SAP APP dispatcher dependency chain Impact chain:

HANA DB not running

↓

SAP APP work processes failed

↓

Dispatcher shutdown

↓

Port 3230 unavailable

↓

SAP GUI connection refused

↓

Web Dispatcher backend degraded

---

# Key Lessons Learned ## 1. Open SAPControl Port Does Not Mean SAP Is Healthy Ports like:

50013

50014

may still be reachable even when SAP applications are completely down. Because:

sapstartsrv can remain alive independently.

---

## 2. SAP Startup Dependency Order Matters Correct startup sequence:

1. HANA Database

2. ASCS / Message Server

3. APP Dispatcher

4. Web Dispatcher

5. SAP Router

If APP starts before HANA:

work processes may terminate immediately.

---

## 3. SAP Router Should Use Proper Service Management Running SAP Router manually from shell sessions is risky. Better approach: - systemd service - persistent startup - automatic recovery

---

## 4. Dispatcher Logs Are Extremely Important The key clue came from:

dev_disp

which clearly showed:

all work processes terminated

without that log analysis, troubleshooting could easily remain focused on networking instead of backend dependencies.

---

# Final Status All SAP services recovered successfully:

[OK] SAP Router running

[OK] HANA running

[OK] SAP APP healthy

[OK] Dispatcher GREEN

[OK] Web Dispatcher GREEN

[OK] SAP GUI accessible

[OK] Web application accessible

---

# Conclusion This incident demonstrated how SAP outages can initially look like simple network issues while the real root cause actually exists deep in service dependency chains. The troubleshooting flow that worked best was:

Network

→ SAP Router

→ SAP Instance

→ Dispatcher Logs

→ HANA Dependency

→ Service Recovery

In complex SAP environments, understanding the startup relationship between routing services, application dispatcher services, web dispatcher services, and backend databases is critical for fast recovery and accurate root cause analysis. # Troubleshooting SAP ABAP Gagal Startup: Indikasi CommonCryptoLib dan CPU Feature Detection ## Pendahuluan Artikel ini membahas troubleshooting sebuah kasus **SAP NetWeaver AS ABAP di Linux** yang mengalami kegagalan startup. Gejala awal terlihat seperti `sapstartsrv` mengalami timeout saat menjalankan `startsap`. Setelah dilakukan troubleshooting lebih lanjut, ditemukan indikasi yang mengarah ke **SAP CommonCryptoLib (`libsapcrypto.so`)**, khususnya pada proses **CPU feature detection dan initialization**. > **Disclaimer:** Seluruh hostname, SID, nama instance, username, dan path environment pada artikel ini telah disamarkan. Artikel ini dibuat sebagai dokumentasi troubleshooting dan bukan pengganti SAP Note atau rekomendasi resmi SAP. --- # 1. Gejala Awal Saat startup SAP dijalankan: ``` startsap ```

muncul: ``` Checking db6 Database Database is running ------------------------------------------- Starting Startup Agent sapstartsrv FAIL: Timeout Startup of Instance Service failed See /home//startsap_.log for details ```

Log startup juga menunjukkan: ``` Starting Startup Agent sapstartsrv Startup of Instance Service failed ```

Pada tahap ini, indikasi awal terlihat seperti `sapstartsrv` gagal dijalankan. --- # 2. Memastikan sapstartsrv Berjalan Daripada langsung menyimpulkan bahwa `sapstartsrv` mati, proses diperiksa: ``` ps -ef | grep -E 'sapstartsrv|sapstart|' | grep -v grep ```

Ditemukan proses: ``` ... /usr/sap///exe/sapstartsrv \ pf=/usr/sap//SYS/profile/START__ -D ```

Kemudian port diperiksa: ``` netstat -lntp | grep -E ':5[0-9]{4}|sapstartsrv' ```

Hasil menunjukkan `sapstartsrv` sudah melakukan listen. Artinya: > **`sapstartsrv` sebenarnya sudah running meskipun `startsap` sebelumnya melaporkan timeout.** --- # 3. Menguji sapcontrol Secara Lokal Untuk memastikan komunikasi dengan `sapstartsrv`, digunakan: ``` /usr/sap///exe/sapcontrol \ -host 127.0.0.1 \ -nr \ -function GetProcessList ```

Hasil: ``` OK name, description, dispstatus, textstatus, starttime, elapsedtime, pid msg_server, MessageServer, GRAY, Stopped, , , enserver, EnqueueServer, GRAY, Stopped, , , ```

Ini membuktikan bahwa: ``` sapcontrol | v sapstartsrv ```

dapat berkomunikasi dengan normal. Jadi masalahnya bukan karena `sapcontrol` membutuhkan SAP Router. --- # 4. Apakah SAP Router Dibutuhkan? Untuk command lokal seperti: ``` sapcontrol -host 127.0.0.1 ... ```

**SAP Router tidak diperlukan.** Komunikasinya terjadi secara langsung dengan `sapstartsrv` pada host SAP. Secara sederhana: ``` sapcontrol | | Local connection v sapstartsrv | +-- Message Server | +-- Enqueue Server ```

--- # 5. Instance Berhasil Dijalankan dengan sapcontrol Instance kemudian dicoba dijalankan secara langsung: ``` /usr/sap///exe/sapcontrol \ -host 127.0.0.1 \ -nr \ -function Start ```

Response: ``` Start OK ```

Kemudian status diperiksa: ``` sapcontrol \ -host 127.0.0.1 \ -nr \ -function GetProcessList ```

Hasil: ``` msg_server, MessageServer, GREEN, Running enserver, EnqueueServer, GREEN, Running ```

Ini menjadi temuan penting: > **SAP instance sebenarnya dapat dijalankan melalui `sapcontrol`, sementara mekanisme `startsap` sebelumnya mengalami timeout.** --- # 6. Investigasi CommonCryptoLib Selanjutnya dilakukan pemeriksaan terhadap SAP CommonCryptoLib. File yang relevan: ``` libsapcrypto.so libslcryptokernel.so ```

Informasi library dapat diperiksa menggunakan: ``` sapgenpse cryptinfo ```

Pada library yang digunakan instance ditemukan informasi seperti: ``` API-VERSION = 1 VERSION = 8.4.41 FILE-VERSION = 8.4.41.0 SELFTEST = OK CPU-FEATURES-SUPPORTED = AES-NI,CLMUL,SSE3,SSSE3,SHAEXT CPU-FEATURES-ACTIVE = AES-NI,CLMUL,SSE3,SSSE3,SHAEXT ```

Yang menjadi perhatian adalah: ``` SHAEXT ```

termasuk dalam CPU feature yang dideteksi dan diaktifkan. --- # 7. Ditemukan CommonCryptoLib Versi Lain Pada server terdapat CommonCryptoLib lain yang digunakan oleh SAP Host Agent. Misalnya: ``` /usr/sap/hostctrl/exe/libsapcrypto.so ```

Ketika diperiksa: ``` /usr/sap/hostctrl/exe/sapgenpse cryptinfo ```

hasilnya: ``` API-VERSION = 2 VERSION = 8.4.47 FILE-VERSION = 8.4.47.0 SELFTEST = OK CPU-FEATURES-SUPPORTED = AES-NI,CLMUL,SSE3,SSSE3 CPU-FEATURES-ACTIVE = AES-NI,CLMUL,SSE3,SSSE3 ```

Perbedaan yang menarik: ``` Instance CCL: AES-NI, CLMUL, SSE3, SSSE3, SHAEXT Host Agent CCL: AES-NI, CLMUL, SSE3, SSSE3 ```

Pada library Host Agent, `SHAEXT` tidak muncul sebagai active CPU feature. --- # 8. Memastikan Binary Berbeda Untuk memastikan library bukan file yang sama, checksum dibandingkan: ``` sha256sum \ /usr/sap///exe/libsapcrypto.so \ /usr/sap///exe/libsapcrypto.so \ /sapmnt//exe/libsapcrypto.so ```

Ketiga library instance memiliki checksum yang sama. Namun ketika dibandingkan dengan Host Agent: ``` sha256sum \ /usr/sap///exe/libsapcrypto.so \ /usr/sap/hostctrl/exe/libsapcrypto.so ```

checksum berbeda. Jadi terdapat **dua versi/binary CommonCryptoLib yang berbeda** pada environment tersebut. --- # 9. Pemeriksaan CPU Feature di Dalam Binary Binary CommonCryptoLib kemudian diperiksa menggunakan: ``` strings libsapcrypto.so | \ grep -i -E 'disable|force|cpu|feature|shaext' ```

Ditemukan beberapa symbol/string seperti: ``` CPU-FEATURES-SUPPORTED CPU-FEATURES-ACTIVE CPU_SUPPORTS_X86_SHAEXT X86_64_SHAEXT_SHA1Transform X86_64_SHAEXT_SHA256Transform X86_64_CheckCPUFeatures X86_64_EnableCPUFeatures cpu_features_supported cpu_features_active ```

Hal ini menunjukkan bahwa CommonCryptoLib memiliki mekanisme untuk: - mendeteksi CPU; - menentukan CPU feature yang didukung; - menentukan CPU feature yang aktif; - mengaktifkan CPU feature; - menggunakan implementation SHAEXT. --- # 10. Eksperimen Menggunakan GDB Untuk mengetahui apakah fungsi CPU feature benar-benar dipanggil saat initialization, dilakukan tracing menggunakan GDB. Breakpoint dipasang pada: ``` X86_64_EnableCPUFeatures ```

Saat `sapgenpse cryptinfo` dijalankan, breakpoint terkena: ``` Breakpoint 1, ... in X86_64_EnableCPUFeatures () from /usr/sap///exe/libsapcrypto.so ```

Kemudian terlihat pula pemanggilan dari: ``` libslcryptokernel.so ```

Hal ini memperkuat indikasi bahwa **CPU feature initialization merupakan bagian dari proses initialization CommonCryptoLib**. --- # 11. Percobaan Menggunakan CommonCryptoLib Lain Untuk melakukan pengujian tanpa langsung mengganti binary production, library lain diuji menggunakan option `-l`: ``` /usr/sap///exe/sapgenpse \ -l /usr/sap/hostctrl/exe/libsapcrypto.so \ cryptinfo ```

Hasil: ``` VERSION = 8.4.47 FILE-VERSION = 8.4.47.0 SELFTEST = OK CPU-FEATURES-SUPPORTED = AES-NI,CLMUL,SSE3,SSSE3 CPU-FEATURES-ACTIVE = AES-NI,CLMUL,SSE3,SSSE3 ```

Pada library ini tidak terdapat `SHAEXT` pada active CPU features. --- # 12. Mengapa LD\_LIBRARY\_PATH Tidak Berhasil? Salah satu percobaan adalah: ``` LD_LIBRARY_PATH=/tmp/ccl8447 \ /usr/sap///exe/sapgenpse cryptinfo ```

Namun `LD_DEBUG` menunjukkan: ``` calling init: /usr/sap///exe/libsapcrypto.so ```

Kemudian binary `sapgenpse` diperiksa: ``` readelf -d sapgenpse | \ egrep -i 'NEEDED|RPATH|RUNPATH' ```

Tidak terdapat dependency langsung terhadap: ``` libsapcrypto.so ```

Dari string binary ditemukan: ``` Using -l parameter to load CommonCryptoLib Using SSL_LIB definition from environment to load CommonCryptoLib Using SNC_LIB definition from environment to load CommonCryptoLib ```

Artinya CommonCryptoLib di-load secara **dynamic loading**, sehingga untuk pengujian lebih tepat menggunakan: ``` -l /path/to/libsapcrypto.so ```

--- # 13. Segmentation Fault Pada salah satu eksperimen perubahan library environment, proses mengalami: ``` Segmentation fault ```

Hal ini merupakan temuan yang cukup penting. Dengan `strace`, terlihat proses membuka: ``` open("/usr/sap///exe/libsapcrypto.so", O_RDONLY|O_CLOEXEC) = 3 ```

Sedangkan GDB sebelumnya menunjukkan bahwa fungsi: ``` X86_64_EnableCPUFeatures() ```

dipanggil pada saat initialization. Dengan demikian terdapat indikasi bahwa **CommonCryptoLib initialization dan CPU feature handling layak menjadi fokus investigasi ketika proses mengalami crash**. Namun: > **Segmentation fault tidak otomatis membuktikan bahwa SHAEXT merupakan penyebab langsung crash.** Untuk menyatakan SHAEXT sebagai root cause diperlukan konfirmasi dari SAP Note/KBA yang sesuai atau SAP Support. --- # 14. Verifikasi PSE PSE SAP juga diuji menggunakan: ``` sapgenpse get_my_name \ -p /usr/sap///sec/SAPSYS.pse \ -n all \ -v ```

Hasil menunjukkan: ``` PSE (v2) open ok. Retrieving my certificate... ok. Getting requested information... ok. ```

Dengan demikian PSE berhasil dibuka dan certificate dapat dibaca. Hal ini membantu mempersempit investigasi bahwa masalah bukan sekadar PSE tidak dapat dibuka. --- # 15. Recovery / Startup Berhasil Setelah `sapstartsrv` tersedia, SAP instance dapat dijalankan menggunakan: ``` sapcontrol \ -host 127.0.0.1 \ -nr \ -function Start ```

Kemudian: ``` sapcontrol \ -host 127.0.0.1 \ -nr \ -function GetProcessList ```

menunjukkan: ``` msg_server, MessageServer, GREEN, Running enserver, EnqueueServer, GREEN, Running ```

Artinya SAP instance berhasil running. --- # Root Cause Analysis ## Problem `startsap` mengalami: ``` Startup of Instance Service failed FAIL: Timeout ```

## Findings Ditemukan beberapa hal: 1. `sapstartsrv` sebenarnya berhasil running. 2. Port `sapstartsrv` berhasil listen. 3. `sapcontrol` dapat berkomunikasi melalui localhost. 4. SAP instance dapat dijalankan melalui `sapcontrol`. 5. CommonCryptoLib digunakan oleh SAP untuk fungsi cryptographic. 6. CommonCryptoLib melakukan CPU feature detection. 7. Ditemukan CPU feature `SHAEXT` pada library tertentu. 8. Ditemukan CommonCryptoLib dengan versi berbeda pada environment. 9. GDB menunjukkan fungsi `X86_64_EnableCPUFeatures()` dipanggil saat initialization. 10. Dalam eksperimen troubleshooting ditemukan **segmentation fault** pada proses yang berkaitan dengan loading/initialization crypto library. --- # Possible Root Cause Berdasarkan temuan tersebut, kemungkinan masalah berada pada area: ``` CommonCryptoLib | v Crypto Kernel Initialization | v CPU Feature Detection | v CPU Feature Activation | +---- SHAEXT ```

Kemungkinan yang perlu diperiksa adalah: **compatibility issue atau defect pada CommonCryptoLib saat melakukan CPU feature detection/initialization pada environment tertentu.** Namun belum dapat disimpulkan bahwa: ``` SHAEXT = confirmed root cause ```

Tanpa SAP Note/KBA yang secara spesifik menghubungkan defect tersebut dengan versi CommonCryptoLib dan environment yang digunakan. --- # Recommended Troubleshooting Untuk kasus serupa, lakukan pengecekan berikut. ### 1. Cek versi CommonCryptoLib ``` sapgenpse cryptinfo ```

Catat: ``` VERSION FILE-VERSION ```

### 2. Cek CPU features Catat: ``` CPU-FEATURES-SUPPORTED CPU-FEATURES-ACTIVE ```

### 3. Cek SAP Kernel ``` disp+work -version ```

### 4. Cek binary CommonCryptoLib ``` sha256sum libsapcrypto.so sha256sum libslcryptokernel.so ```

### 5. Pastikan library yang digunakan Jika perlu melakukan testing: ``` sapgenpse \ -l /path/to/libsapcrypto.so \ cryptinfo ```

### 6. Jika terjadi crash Gunakan tools seperti: ``` gdb strace ```

untuk memastikan apakah crash terjadi ketika CommonCryptoLib sedang melakukan initialization. ### 7. Cari SAP Note/KBA Keyword yang relevan: ``` SAP CommonCryptoLib segmentation fault SAP CommonCryptoLib CPU feature detection SAP CommonCryptoLib SHAEXT SAPCRYPTOLIB X86_64_EnableCPUFeatures SAP CommonCryptoLib startup crash SAP Kernel CommonCryptoLib segmentation fault ```

--- # Best Practice **Jangan menjadikan copy binary dari SAP Host Agent sebagai permanent fix.** Walaupun binary tersebut dapat digunakan untuk eksperimen dan menghasilkan behavior berbeda, versi CommonCryptoLib harus divalidasi berdasarkan: ``` SAP Product + SAP Kernel + CommonCryptoLib + Operating System + CPU / Hypervisor ```

Jika ditemukan defect, gunakan **patch/package resmi SAP** sesuai SAP Note/KBA yang relevan. --- # Referensi SAP menyediakan dokumentasi mengenai CommonCryptoLib dan CPU feature information: https://help.sap.com/docs/SAP\_SECURE\_LOGIN\_SERVICE/c35917ca71e941c5a97a11d2c55dcacd/1e8809d9c0094c3ebcdfa981fb858a67.html SAP KBA terkait CommonCryptoLib: [https://userapps.support.sap.com/sap/support/knowledge/en/2543666](https://userapps.support.sap.com/sap/support/knowledge/en/2543666?utm_source=chatgpt.com) SAP KBA mengenai tracing CommonCryptoLib: [https://userapps.support.sap.com/sap/support/knowledge/en/2491573](https://userapps.support.sap.com/sap/support/knowledge/en/2491573?utm_source=chatgpt.com) --- ## Kesimpulan Kasus ini awalnya terlihat sebagai **`startsap` timeout**, tetapi troubleshooting menunjukkan bahwa `sapstartsrv` sebenarnya dapat berjalan dan SAP instance dapat dimulai melalui `sapcontrol`. Fokus investigasi kemudian mengarah ke **CommonCryptoLib**, karena library tersebut melakukan CPU feature detection, termasuk SHAEXT, dan terdapat indikasi crash/segmentation fault pada proses yang berkaitan dengan initialization crypto library. Dengan demikian, pada kasus SAP ABAP yang mengalami startup failure dan segmentation fault, **CommonCryptoLib version, CPU feature detection, SHAEXT, dan compatibility antara Kernel–CCL–OS–VM CPU layak diperiksa sebagai salah satu kemungkinan penyebab.**