From c5401eb33ce2da838d49c86a5ca5470f73497e3d Mon Sep 17 00:00:00 2001 From: SamoilenkoVadym Date: Thu, 6 Nov 2025 10:47:20 +0000 Subject: [PATCH] =?UTF-8?q?fix:=20=D1=83=D0=BB=D1=83=D1=87=D1=88=D0=B5?= =?UTF-8?q?=D0=BD=D0=B8=D1=8F=20=D1=81=D0=BA=D1=80=D0=B8=D0=BF=D1=82=D0=BE?= =?UTF-8?q?=D0=B2=20=D0=BC=D0=BE=D0=BD=D0=B8=D1=82=D0=BE=D1=80=D0=B8=D0=BD?= =?UTF-8?q?=D0=B3=D0=B0=20=D0=B8=20backup?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Исправлены критические проблемы и добавлены улучшения: 1. **server-full-report.sh**: - Улучшены Slack уведомления с детектором проблем - Добавлены автоматические рекомендации по исправлению - Добавлена цветная индикация статуса (good/warning/danger) - Улучшена структура уведомлений с приоритетами 2. **generate-summary.sh**: - Исправлено дублирование контента в отчетах - Удален незакрытый heredoc, вызывавший проблемы - Добавлены правильные разделители секций 3. **backup-full-enhanced.sh** v2.0.0 → v2.1.0: - Добавлен полный auto-discovery для всех типов БД - Добавлена поддержка MongoDB backup - Улучшена детекция PostgreSQL/MariaDB через образы - Автоматическое определение пользователей БД - Удален hardcoded список баз данных 4. **health-check-alerting.sh**: - Добавлена проверка наличия 'bc' перед использованием - Добавлен fallback на integer comparison без bc - Улучшена надежность проверки R2 storage Slack уведомления теперь включают: - Автоматическое обнаружение проблем (unhealthy, down sites, high disk) - Конкретные команды для исправления проблем - SSH инструкции и ссылки на admin tools - Цветовую индикацию серьезности (danger/warning/good) 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude --- opt/02-core/presenton/app_data/fastapi.db | Bin 0 -> 172032 bytes .../presenton/app_data/userConfig.json | 1 + .../scripts/backup-full-enhanced.sh | 467 ++++++++++++++++++ .../scripts/health-check-alerting.sh | 377 ++++++++++++++ .../scripts/modules/generate-databases.sh | 2 +- .../scripts/modules/generate-summary.sh | 5 +- .../scripts/modules/generate-websites.sh | 2 +- .../scripts/server-full-report.sh | 125 +++++ 8 files changed, 975 insertions(+), 4 deletions(-) create mode 100644 opt/02-core/presenton/app_data/fastapi.db create mode 100644 opt/02-core/presenton/app_data/userConfig.json create mode 100755 opt/05-backups/scripts/backup-full-enhanced.sh create mode 100755 opt/05-backups/scripts/health-check-alerting.sh diff --git a/opt/02-core/presenton/app_data/fastapi.db b/opt/02-core/presenton/app_data/fastapi.db new file mode 100644 index 0000000000000000000000000000000000000000..b2e5261ec3ce68921dc113e37e29570e97a6e059 GIT binary patch literal 172032 zcmeHwTaX;rc^<$UK#CMZDH5V6j#?{H1jyaRTz6)`F!EvnQIN0z1b`w{6l&8m)3e*c z%=B=&2a6Spj6h13TyiX1_Jb;yiv5s^^WZ94HZAH#S0yhlKP?`evR_x{~k0xPx^TLz+&s*V@Lk?flu6j_`qX-edO1V{`bee|7ias z8;>>~dHmt4hogs|KJ?EIAAjgiAAIDYFWnzJ@b@TY|5ya>Is)D&4jpZL`ryq6{eIiK z;on#byMEgX*9Jo`^!kw-`9WVE{iBsjXO^#=an7DUedZ-6zoc{Fyd$NZV`dG<-?;Yp zp`)Mv^ug~HBe&W0q>TLiaaHe?M=jNS9 z9yaTZOVAHpcPki0YwN>cG;qGYe2LypyeS28f12ar&tY&^&OCo6`C2RJM}Wfl%H<2^ zshy3e*Im;ORXg-N9C+?EZ@AVEAXtp&gJCf6hLP`uaTOOYon2kN^bO~$XTISa^V`Si z^)Fqxbmr{y=jmAzJLl4wFP*t`=KRW;%Z@qtkY6RpPoFt=1~ap=e0gR0^qEJFKlAa2 zj(+je2M_R|!dJSGbZhP?3i!L8;iAC*i8mTY4jsLG`QV%1;Hh!Lt$r&p{cG!9-y1R| ztVM2kEu5jq#}bn|!wb$ap7FeC3qggWP$Qa{(ViEE?s@`kZ#WEwG0?4{=SE(84Kn^T zi1*6b)ia3-MuYZ_zvEXot}H!kj~w?t_R!I-%aeefVc2WM845jq^U;qUI(oKt@aB`6 zOE_7)o=W^K1C6ds?o00yBSdE9FA0+4@JH=-od+6xZ+N3GY9m85N z_wh$>mLEBE^s}Em_(qj+i@e^T3+fBw4I0g^g=VS)FWx~omes< zxrmY{mQ1fuR}LRK`o%9E{BDgWXX0xUe|R`Cc@vAN$;?}5%y;6IEitJIIV_ENf>QlD zkt7VYMsAgnLkZx< z&vzpGX&v>$wb7s(xNWcPoVsw~+?nO`2_Oeh9(nOywKra!Ar=L5==J06l z_)X{jLq`|TtjOtfvGn%`67!J$b*I_j=5pcWLGG}Ce(`~aj$U4vgl4)AYsK`F2j3hX zJaqJ_rw)Gq^BJ}cO@0dRON=h@(@sN6?eh{u)K^=n++AbW?XQoFy-DKD$d9_Q*k(5L zyWScR<(Txam+ks}W79Z|=BF{Iuzx%mVKf}IqS4TcpZI-wMq7S8o7A}|=qEaR-5WN8 z(2usvLVmy19ko5|5V0j#3p#p-9BTffn&LtL#-Kl--^j{NMS|L@3;Kl-;vzI^2Gj`WUv z{)0R|+jonAMZh9p5wHkY1S|p;0gHe|z#?D~*hdIFasTSUTD?`Pd4+1B-Yi!tOGVf7 z8jY4$F4fyDuURM_rb3lM2Tux1oob_1skf_ct6VJB>cwWGSuMG4`4AOq7n<#6rPL^v zTjffp*=%%*^`%a`vQ%p>)t9`^LsY0yDV3|GTB+2i)+_B=vtDZB_d=sxtd=_M^3sD; zsMV?z-D0&;!;?z0RHzr~wT4$~*IU(kwbN)kz^;)@BYA!X36}M7oRGMzBSZI|C zwL*Josd+yYD!a{AwX#(9s?}Ph)@U>;t(NPR+MRl{)o3@p`#yjF>ceiO;}$B#W~bCC zSE|)Qqh4^EwN9sus#@)a_ZX4H$Mz9j*r8ekECLn*i-1MIB481)2v`Ix0u}*_fJI;* zB0&0od;Z^t6K{uX5wHkY1S|p;0gHe|z#?D~un1TLECTxq0ek-6SJQ0=YZ0&rSOhEr z76FTZMZh9p5wHkY1S|sk6aj1hzfULL4%;GN5wHkY1S|p;0gHe|z#?D~un1TL_7wtF z|G%%M+YZ(uU=gqgSOhEr76FTZMZh9p5wHkY1okNcR{y_GC*BU*B481)2v`Ix0u}*_ zfJML}U=gqgSOoSJ0#^UOucq4$)*@gLun1TLECLn*i-1MIB481)2v`L6DFRmizfULL z4%;GN5wHkY1S|p;0gHe|z#?D~un1TL_7wtF|G%%M+YZ(uU=gqgSOhEr76FTZMZh9p z5wHkY1okNcR{y_GC*BU*B481)2v`Ix0u}*_fJML}U=gqgSOoSJ0#^UOucq4$)*@gL zun1TLECLn*i-1MIB481)2v`L6DFXKVzfULL4%;GN5wHkY1S|p;0gHe|z#?D~un1TL z_7wuw{(oOhw;ilSz#?D~un1TLECLn*i-1MIB481)2<%e?tp0zWPP`qqMZh9p5wHkY z1S|p;0gHe|z#?D~un6oc1W5n?z=NMY@X?Pw>Kyvd_+bB71S|p;f&Gra_a1xTBde>Q zdG6sKTm}@KLaR~kG&*jx-Kf+$^?Ik^b-a49?ABZ5(o&;R=y>H)VX0GXv?}#>)oqoF z#ag}CY&5GS*DbGmeQ)S?)jubjZs@lb{C+1`2)lmUdvz}Iqpml1(wVzB4BDesuv8(5E67Mnh+yiUpmHHw>NDD2#%h|82L~^_;dFZZrdT*bbde zFm#5V+g*tKo`+9^U>N!Rb;s?uQ7m#7+C%@k*LT`p%MVd~xagdX9KRoV>q9s4LTA&D zHk@d~AGQ|;?l9VNq9Evo_}lUN8*ablIl%xUqppQqYuIwT`n~2T^!r{II_tw=6ZKGU z{2sb~-D^8ur{lMLuix5Qq%rzEcbx!ub#85qKd+63-L*9u??x02!jmUX@Z)fC)DH(; z3~{j)^iB*mf+$!hR;#6@rAn<_SZJ3EwU%42R~ohY7yTPuzxmv-xlmawESA6E-|*Yd zt(G@$tbS{`+&O)!zWU`Go9^?CLUXIQ`HdIb-OiV9)H~0AeMG=nEY`dagAL24C2Gu2?7(zVOO(^#YAR_Z#Eu4I=6KDUCqnxjo?O^gNFzU>F2F zr|pNBIAF9M&u7PTqtOrm(hP+`cf`ofz4qD&Fkg`RPw~O2wu2QnB9Y z6gm~RTyJy=Wv^OmE>${3w^FL!CPDVRXy~`X3^lH}Q8@2h^yqKu(M4~_C}84GI$!m+ zoXZhP(CAeSn6H;#o$IKo_ugU1*&MsB+db->UMv;TxoS@^x?{B!* zJ+9$d=k?0cs>9!bLg*ZOyX!di>177qxrZzdJcZXs6|LPqX98F-gBB> z6nP{@bYnya#R8b_Nv9iZ;-@eI#ezb@(Lf8B)~HK#%tRAGBJ?8%5P@So!uu+-3r6l< z^M-5vAo7G=Fyqd3KODIbF)bHjiKLU7VrRo&-#}xS_h-Q_elYYS|Jz=A(ifcUB#^eb_I-0ys1#lL zyX?{5l{$Ye@aG!;-Qp*8hrd+$Z~j?cD7WQj{#z=lJ1?DmiHny@{Fq-V%a1I#!%;I{ zeV=~1ST7X6IVu&33v+@@}?Qx_Yj(Gp@QK{gGzmZ|-_|iOa}fWOya1TSi%v zFY>kw6CD8*@`gO;_sZ%$`KMx(DmVEJDaBY+nu`t%s#JRVX*7UwRiqv1lhmpPhkefH z|7duSmYJ^}{>9}3-}~YxA%+UA&QhVVRA^R;FVt%d55&=`)~lUHBfakdMM37!reeW|XwEJ;)c>NODHusV1xH4uQkDTsnGvA` z!LG<}RLWGjg3D+^(*i>$b0L%>FRQ7_S)aJC(vI{`73GTLi9wH*&b)M7Dwl?M?i9sH zb3!HZtB`qF4V!x}6UNhAk;xVuF~=qf*3tf133F4;<+o zIP!N#es-jHD{hySV#4fB)*k(=Kr0^Z$==`o|r>oea;uYZ0&rSOhEr76FTZMZh9p5wHkY z1S|p;0gC_-pi}!a!#qC!PbO_0!cA^t>Rj^Pjod{d7xpT?h zA=( ze*c`;Uyn9WXPLgcH{$P=0*v!gwdc2H7!Shy??%GbP!A-(&0-3r;SCmk!xm5FI+lD26)m3^e;OEF-N8TjD|khFy>jH!m?OR;*4X* z(5b4S_sYnJSsSW~ zGX1f|Yry^kFr3!*j$*d$_s>RNkHmQiEkX3^uOx}J>~M!emwClkWyfM=qJ*2cHYg@! z?vXWO`m&(<7B35R7SMVmd6-;?hr0ovhj`EqTo9Ttk?uI&4R{gs*>hof`&GdvZeOnu z5G35V25KT2koPDX`!r;xP>Xx2&;VWuZIZ~UhF3=3aEq;_$8(S`+S7e51mSW4KKCtsG_ndgcyNvuf;LWHfFS15TCV4x?3zQ^5@!z`-GAzji*l7$xT&ZqXz2JI_9`VeD)+kU!=WXv9418~D8FKTQTF8Z;GCTumLr9*UE`VKc!0Rw@S7`LhK(*{{XH=R;+ z$(*T&Pbzta^3~G}Q&ysDHeKCr=IjR3gdxd@Mqh%=3WjYTCpphtOEubPb1Uz?Z%lbF z1M&^3eQQ@o8oGg-d3u-wSPtiP`{EEsis6x6W zW0s-X7dJfYt!YQ?P(igmbfvs8X-)Y4dv2d6oC;pS_sAd2UL^$&pcf|8DNEq4?m6Ox zi&rdK%4pI5Anz*WiX%zwnVH+mU8TD8w!6wnR~qJAfP(jWS2bJMb6S==Q4#TVF5PY(tzi2Q{EzRF!{dR0F1Xl z^EkeHA;c794Fx~hclY83AbCPF{{{pMsbgm7)pNWUo4T_}`^6aa^P&dHND+Fj^>lGz zQ%L8xm%fx00ir74i&gBVB&JJp;B)jm)gYI5dWd6#fD^n+?ot4nhz=Aup!nDDFlf75 zPlfzsx4pBPiT6RL7t<6IAmmGiM(|8#W;a4C5xm9}U#k+;CRcRH?X6PsL6yoopBCaW zA=t__1h$D|7pdCi6wJXv9NAk+Ad#4|5vYXK&U|KGbgs~O7{Ythg~||`MxyFbh@&;% z{OU?+qoQhif6mD`|NrBOqKMC_nooDBz(84vFHf^p)ZQ8mH8JIB1m^~HGyx7N?hJB{j-^Uf2cRfGyU zA0XHcXp>7JOxIUEZ-Dqfy#X{4LP1PGQ&tF?3B$UgILpoDX)nT64k$M!b=({&s}7mg zMd!>|nU?x#x}eTowuojDiY1vKJQW!6!RXGePGuZ%jQ~+n;!Z@5=$7bOFl%%m)2pu0 z&g!c}v-cCO?u>9sEFAUiTYOu`h?KnoUGv3Pb}QJgX-VXEu@NH;=yebB ziRF^?l{b9vk37l~5O>#@==6A#yT|u-cJHxwdk=$eb1pP?x1{W`&gnf}Qq;0~TWd^8+BiZ9c)f9=CUtt$xqNmrLJ>bgH=(QK*sGOr-InUR! zSl;9=Wmdh4>;#nrPD^SL!@6y)P2p>0wGJ3#{qQRMt-#w;ftR@L*+J^fN_Vt1Gj`N_ zEdGS8bGj*frj#r(^Y1K)(mU$+<8pL1!8Kh-O|B=Fc!`;wUSn45y)r^f&Foh6YxH>f zahSJ)uCw8}(5tM^WQgF)ksy%H7}UVHqb9LP%m* zHs&?A%)jU%Nw4riE{Ea#e;~GMOmmrB=r!X>KFf!fJeH1&C_11E6*_ff@SLb{)K*CQ z)*xO}DrPF4GBw9|l%9m$2!)8uF>uwdqmGjx&^P^8IC3~M{#g{5Sw^t@b`Z9gW*ZiH zuzrk&h+mG_zX)o~m69h2F@WE0lM#KwPF`P{sS27wgfQkG+LI^afxB-pZZNih;Nm%% zffzjzr9>1I-()sMlxtj>=Qp03`&p-NlTC7WoIL5m*s+B1@)zhgXL!D_QB(udr&{FnN2{^M@?05p8_0kw=2jFU#$)z@khCI(AR-;)3}iaN<(1}dRi-bGm18kN%_gQ9 zOR_IU9csR)50jT;e3UuJ7$%Ho@ko=>Mrq(aA)wFbIg^KIIf1-g3oPXl?o?a^BPhy;~}HB3KNC>zCWz- zu?aM6&bu)^LC{LlFcRL%%VQK0f`aHfK)volNTVvng1lhnu#jFF85k;Mo#%kNmQDB|BW`e7190H97A>M-0C$bfXh>wHE712Qb0;vhk>66;vQEHsqUCJ_T^u79zcpBA z+ug+VqIeGI^kalZ@iuZ=82A)inJy-Xvf#&0RH6z(snBfl8x$tPhVh>6vUIk>waYY0h0?Z+MG~|~ zb{QdPm5fG!5V|#pDlv8$Rw9>-N!3iXiNqyFx{rz?EE$&hngN#Tk)R=93DQEqo{UgN zC3GyGxt)rUjie%;JZ$xMu_wXr|)oF-5Q34!>?3EHG0S<=MQ@I(_td4dRffj%vU z*Fl^RhHy?u5TigXy%j9Yks1kBF#VGqk|;$MjDp%i31cayRdVQ#1ct(m1tFjZoeR)14U$kw<kk($e<8mRC=X_5PJ`#Tcufm+46XX>-rGu2lasTvoL*Wfe-Z&~$YFBT(i zPeEsZW9EmG9c9H#@TNKD zH&Eo)+wb7TTi?aUU*N&JsPWzHpDsGv|9<;ze0u|bOA$2l!|k8pe^g2DI#k#8pHZ1x zZ=(FaL(AVmwOr-)+t>;H!*~7w2YYAx4{rSywTWuB|5Ck+-oAmB-p0Q-@$Mg@&0lW+ zISTxod$;}Td54?+D?EM|%N(l08uwq`TA?9s|LWF{P#6QGp8o)~e244%9cTM5)NpTo z|JIx6Fj_%(-*wQro9I5{m;Q)yvUnEeG`>`*^U!_;(EVS!TCXs4^%~I*Ht7BpksIG{BE#g zv#DlK78>JVS;j@E>a$j_&Zw81coV13N>lrof+|0*;?8hqbc9G&On{GXGxYltyPLkP z$m{l=#5K&^Z|B*rCVPXC=m+s_3~k%)c3i``O|JFme>)OH1_$ejSFH_q0lg@?F&^Zguek+1Vc2sduZbTbH59Xo~3`t=)C=P5a zvALzYz5|_sKkJ00p+yxNVDinoqVBKr_{9`}GrEP{Tlrz#2yBuTccPlLg zelgKAwyr4$fjiaVkA0W40LmAHX(K{j;wYrlEJuX&0$C~%K=k%3r-LlM^mdZCr_FNi zu9>Py?9yVdRvB2_$5eAy@qOq0Xr|^^f{aO%YS&bT&xWMXGe5nZu_2Lc{KOSmxmhiU ztJ8uQ2$_4p%i^)Dd_YvFpVT0s@=N@#Bm{9)(s~%01H5`K3WxOKU0$MjATvd5xWb4)oofX5jRj;jL_1g%Om|o*-BXlpaQF;R{v?a$>n%T)5z*1Ke?rtp1L~KqcpL7>x5x>19BE8IDn3GA+ zta=uDi-~jkSaMi-azFOqvVg6TgW(DVI5iguMf5`_{>26n&)?7lx_6ch9mx&b&bABr%z`4au%2 zvRCW{i0ze68XgRl@zcw53@uF%hbWu7N<<+(B$+*;+Hajzc>-o-!`XIz`UtGuM<%mB zuPb-L{w=TD&Gk{aMDY>n$7f-sJnG}3Nn-qnXNp1v0d+Z_4RA90xTn~pWd{3eWVls^ zF_V~(xiu+aH}?TC%~0%E3_rnZ^nkdH{-wCFOh~w}$OL;;LN%r$Lhepxa~z66*D@Qy z75TJ~J*a>&D3mLDh3Qp}3u;2N8$*HrWaap6$4elrB zST=R6yG@qm5ps}1ijX9gYmnnBvkYBw3G|vLKRv?4Xz0hd=lEhYHI=MUT<^?Gy3U{b z93h+SHlOTKttor)m4zt&RqRiM!G2!!NH1=>CsTWkR+D?gD}K+@Y!mMbjb~GrXDQdjTAonQX#itcVyWXj&`EdptLo%mW z;7)Hgw%~&|Ll@IqM?NExx`XnfO()p&>m|MznepJ2y_HaVYi$Xod3fR?pzQUuac5@# zLCmAdjBWZdrBbt&Ue(~Rm0`j}!`Vh=;jiS~3r`m+($M20qlkZJjPZ(+n!Tsb+ieM& znBu83Ou9x}(}$Wg{|CiXuD;e8_4`{m$g=(aV>tgmaOh7D9C`Gk{$u|UAM78CfJML} zU=gqgSOo4Z1aAKDUq0}u)fXOq?hXI(-(6i^e&fw2pE>ZE-#C2uaH|C?@?y18>$J<2 zW~opw)N2i|)~>gz^=hZl*!~YYKKASoAiV#b=cJP zJFqwXp<~*xidn2-!|I@!DMqXr+gdh+#e08@&ion<9AxtP@>SE}@l_njBzigyMhk4V zd^@(}C&kd&um|6@@0oJw5Oa6&To4}&`CD8(#EU{X51Iz`@`5sImrkiK+CU$!8}hao zMav6fnJ;GK%HKx}$c+(c3-@pK6+U}sWAk`?GGSu4&e$vW7BTXd`7-BU2C1GiEhJ3%g$fEe-bq3?;vBM9 z&Oq=BsYt!zvLZJ3LPCZ|g7)#G1W(8_UR@{_7Ya*Gp>eWQJXx$PmP(Cksge4nbh1#x zFU4A={Az4Os4`a2y#><#AWjRJba9SpJYnNGx`RS8tW*wAM0Rg2yVa=JNd?Uw!f5a|a&(D}-CD8G zDi>;n_R><5+z?p!-kdD*A2~cp=)ZiG4D#a%foo;?WT|u#{>z10y-})8`lVc=_vrtW zqxntezK^fg4t^#vhO*miRVzznuUf5DYK=yt(rUS0sokkJTa9+p+x{zZh%gr{ct;#Q z1J^!3+{o)%`H%hhtSi{oFGmL_%am4lyHy;)7+>{dE1e6*UKQm0(0Rtt?f z9Gz>OP8mSA+6`~}_pq+Pbz}Qi@NS@06^q)f-x9x$o6h!6wts?!&tb<5S|#7veuup) z;0?j!!fSAgP+lI(u#Q58l<>aWnHG$!P+Ba53~IH^nAI=!>Vz@V`TyX3e{%p@e&8{6FIufEburk^?MlXju>|}S<0p;F&TYYd40WsDbCv8Rrc72Vlm|tZxk>b`fw^y|#*IyT#=$1=K}v=sL%Wq3U)C%X(|k zSvhmuaa)+x&^a?21_SR|Cq#aX08miIqM?J}aR}Fg=$9MFQ@t4syX{T5A|R9;r7v=p z3We2YkbK?`(RA1kf^U;^vEh8aae|5+d%ae8k~BD7zvH1nv^^N1UX(>V5Hrrfm%qK} zTyFUkuWlZOvmP|k{n0v{-wBt@ZmULns-;r7uvCxN3fYI%PnH^sOVvhW$=GeNZQrbW zEiQagf{RE_Te?0E7g*o4U}~wbRnv+rW$E{#4=y@O#hoi`=yzJjHxwg3^X;*HAQK`t zqUj;WMv%c1ofpAq)Wi;Nn+T5b{aP(;azF0C_bub20p>h43w?nprCw8 zEjCq+LtUNGraY|_+zWHGVxB!(jA5QieCmh@>i0@xW-td@Azm5NIUejniR7kPUj)Y8KKuBhP!aG-h5DBn$bMZaRg z!)GnLW|4IIE=u-C>>jr;gV^AN3Nr`{E`punS$aVk`)IR)w7-N%l%NmyS1-h2?4R)Y z|G@t}fd8Jqv%}e6-r4K#$J>7&J@lzn59*YwkN@Z~J~BQ-N5+RAZWo&EW~J09mRsdY zr`c?DiuI*VyMlA$Qhmwmq>hZ{H5DHeonix*K6GTu%9}sugDZ3{ai|fxZTb~dG;eW@xP1C4 z#r1fLqIn=_$qyG%U#aB$!*~AB!QB($cd%mS=M+-pT|WE1b?dir6n=xsZvT9TW9aAk z<~JvNkz7~vrMrab(=p7vDOf`=L3R8+4sY?D9^@emuLQ=}W1V4!%B^^jqAw4soE4Q6vgqOs6QjwU< zs^kr0&MKYb&w`w46sm5l3z#qr6%$NFsWL`p#=$egDe6$7s#h5dRp20nD)*#8+Ou&_ zL=F=JXp#mJ_f`^{)IQ`W9)6N1;G$7N9GxSCUr%RcK?% zi}hjy*EVB?i@gFM^YpomC>n$(Pn@9L_Cnag-Hy8$4A)PzhsXwS0zeiPilr0s16`Qt zZP>((+pPP`^oeeKe79Es1TuorURJl95bE{{fc6}GqMzl?aCcV#IC6g$|7SEJ zq$sl?01f?8mSP+P%@pSokLNnHo>@5dXGV3(ej!hKnQo}*)4S5g+7N)odHut_98d^P zB0w7F+$;*Cq-V6aqYWInI zpjlNEX(=XhaK=a!xe|=*!xHf^<3*5booNx(G(BC2=JbDZ2W$wydqDjqMtn50V7WL< z6iF}lM*NkO7+MWyLjVG@85BRa#R@VDKv}ceEPR<AZed=1oh5$4TSlE-DNG7>V z>X?3XkEA`P%xB4kz=i;fDA`PQz1zWO$%X)Q$jAru#eOB`j%4BrnX$FfyKQyZMYW}T zoy>akuf)=(M|Z{dVBdrQtf%7t;rKsiDw!tE_4qCzC2Lk)`0(ui;+`ehS^TOWyss(o^kS%jzi1z0l2qYB z(C=<#z3!A>@mVV3TtE>b$Q1C)LaOlF-U>!ZZ#xR7O!O??%#Ngb(8&{0Zp6pMtA=|c zUgUcg{beuAi&KN%V}2E_7K!y#JC<{lq08)?0}2&-Q+aW6P}P=RNxC%r;E6UR|AE_$ zgFSN+E#syv6^WiCv6h>6c2Sn10mX#OJ+ekjUl!8p8m#~5p4NTxRY|Iq3j?xhrr{#m zJi;I72-ipj*+e9U4x%j#x^CZP+h=2_`O8w;p&Ln*?{HU?>aG^|p_&93%L={~;~ zPUxAXBEFL64dYb|Ww)8BM24V(1ocm^aePWjEeymD3$i-I>$E;LDVk)K;ng8`VUV66 z_(-&ctW$<9QjfMfYQI6z)RWr-6+{H#uMmerEgC82G^ha)Ixy(>$A8=C2JLh9cvVQ& zo>?2XHKUB|;+Rd5PXT)^+D1ZdB0p=-0UJlbz;ER+3(zxNuQjXkMk+^m+9m0ITat{) zPA^8OllMKn7|99Q7);)Jlvwr5vLIKz%;v8>nJqf*5J~Q`gf)-OI~_J4oDAbyvwiUS zhi>sP$ET5+zH$(#iE~f9n5FB}hq60fnm7@Nghf(|OZL#RLdinpG@*#-RB!{1ZRoAp zFU;c(M;D{yyvlop03;+)$>M_}0w7OG0G|t-ZgBoT&Wlx(8Zr(tj(TV4uWv+)e1bw{ z0R=OQHhcu?>PXrxd~~lP{uL!m-mlBq>G7R~tYpZ9gEUbDc}!Qg%lNYEWTH8(GbFkF ztqUDU97H$O)lC!xs=%vrIWLWc51o!VeKVzN0#l4D&M8AFo2HfFJT=n_A)cE{vB`+6 z_1Gq#PH1zhhr658No=WWFa~XT2v*snIb6hOMh%`iI8OdF`Smz?tL~=C50$-M(GAHP zS5g1%?wL)k$<+7F37t+Uy7xI#A0iSTV%yR5&oX6v^RBDg&74Wn`V`_XVc(6IzdlZK z3B!1eHu@$xM^o>8W6FCpsKjD$g9Pr@3>I*iH+jq&qp8!w9Kf^uqmeWo;mwr>JKN#@^B!MT=yv~Ufl2?mS{&!r)8wF;}xeXHM?qBDSUbVJ(Zoa;yzyeQ}vCPbxfWJ0DLewSAEl5hp!2S znckJ`)rh(xSztey?AG&aKN<$kWdyBd*=lx}x0|;e1)Ty}Lrg)|P}KE(cQ2jDMzw{0%l?981(aq)te`*>ABX^#f7K0m%fyh-Kna8FIKUe%59LIryAt)PH%(& zhfO+7;9!D6_=e3OKKs`wbT5M7J{9tl-S*CECX`pbR^dsiK2VZ=FuP`gHWrC3(j>_i z(-adR-pbGjp2^JYMu;irgsk{lRbC6IJ;{0LlQLAPyz^-x=>skruX$TI1>+;4?jw3K zyFSP%xa$trJxBJIIOCd;gAuoy)y{loMqp<;4?}p55agPakwn#_5XWo2`PG$D4XU<# z=$wr6{{fDW)|N<86ll9pF>}RKvH``a&pJ;(O+mk&ej3Hc!M1KAqLMZ&rc&0(l#~_K8IkDp?*wb5s`4;NBNBd%o08yyD&U`_O;X8o1VAIrkqokog^UT@ z{KXhSqN2exEvwmSvCcmNCAA6XePd4 z#@av^$m5i)b2$4Wm^`iF-@=mZQpUh4)5&j>c z(WsgQ`n(ukU!SAh@~d-Wj;a-JgQIpGNlSRPgftmb@{&ow%E|F8r3(_UGZEd%u_NZ>Az8-hRW=@v?0Pos*!DbZ-6CYb9&s$4ABaHhR`&pT!+@=nG<0uGEqof zCPiepiq0vsWk}z$lHMYJq&0JKkqUB`%HKRd%1Qkx)`!8W$cA0u>6B zCqlk5R>U=U??d{^w6Z8DPFaox9qQCxME9lIio#dI*&_D@27yHSBlMzX$dFlzHHB>C z1))Mg&)kQqqSB5~o=8ifv&5Rtjp}W)R3-h8AA~*)3QBat+?O{H^>_QnW(09$Xyu%* zl!R3xe!q$9jjufxoS2l!AxzW5gK0^#+9w zS3$!$&#SZj`y7<~4N0O!Dest|%#h=F0ge9j)(@o`giA-TbP83z{ckBuITCQu-*2l~ z=QsZX5!L@pqMW1HA8-H3_U|n^+kc3Hh;@z%eu>`^vmAp(>y(U-64+wwoJEil?MWKi zzd&kQ9`7&l;diNdE{>di+dp5VJb82|-Y0)3{{MXk{_BB9{|Em1VE(!JlOu;ewOU2y zw-+A&lNUHE@(Yq8vsQ1_YF?pQs5i@%%2Ltwyhfwtl}q(@%WD>jt)ctvEm|YV&JT&f zHj$G}XP3N!Ysu$nb4gj*C?6YgCLrxvAAyF4C*{h=yxL+LQ*8D^zRw&4gAwkJunWcg zcb`lFH@rSly!ElL;vDtV3OBs6LumcG(L9a&GL8F1jC=7Z=WK)`#10;NF`|Cb!;aj6F<#PdW*m}2I81r7I>tG~xh2_4HKkY2}dGwl8w-aq5i<1Ty zcabQX%>wro^a=YJ_y~LTL?$E2MqcrofSfZc05Cmt|!q7 zlDLMj57_jEol)2MQZQ_JFyHZz*c50HEl>_Xn8O&&Adei57%pj7TcbW(GRAE>r(xT( z;jDCBoGoza?Gt%G6$>2+<-((5HJa`E(U#~+0;EEvt^uK*k6=e+$lQfim)`A#UiUg+ zK;Y{RqAcRd5iR05dP65}Q0x#%%`gRYl2JFE(qwyt`k+83!08Hre0McZ-aVg!L9V zaS5>(6y8W;hkMv_p6`MbnxTH5Rj%GpUG!e}K{H*4$=D(5Cr~B8bO+eOcC!#(0lLdD>H^zqDMFOwd zFoVDZ=OnTvUZ<>tY%4)!LHziFTNWUPTbCj0OfYgH;~w~m|tU$NVoR1J3icX?2 zbl>0{{D=_s#4d>ILmC!7C%|HB=rbF7u&pK2c+}2$65Cz`doE0nX3zmlo{5i{Ygk$i zknfN=`z3G${Xpgfes4g+;JQB~lI1=UH;*`Jp-`V9KbOtB=0DTkzDBg`UB6*MWMK29w@+oUmPKb4YELNS^TTj(QoiDVobTJXib zfGr&#dgve;YWp3eccfg42Ja1!;}gw-RJfKg3-n0jM-UdQA!tJ`g5*e;fX;*d`fvnh zcSJ8s0uRg$rbRd05z2w^Xrcs86pu(ysKk>vKlc(8x(FM+&ZbLmUUu zbJllIm0#x>T4sV(OBw2Kd4mY67@C22YTNk|t%2@!KltXk35tKFHUy-~UfCmyH3~2ZcXjXWRB&BMT@WSkN zMzDcl0Ya1@2!&XJTz1(K(xZXWuRv`B%NUpo(h1rb5;UNb9#*7i3$jPnyzU50f`tbP zA0N3LvUIqjU*Snv;Ux>`BVRLQ9I-EW6jM5`8jTN8AB`*kHO6axdeJ>i3n#b|Ufnj7RsF9b<{FYxatYmaKnwr1@}SRksTjKK+KIHujA^AXRiqXM&t)A7wu%O zdxS6NQdF2%(6|*9{!e3xvk-&`A~n02vd+l_E!} z8_Je}fzHB+o;YXvZIKD*z*Ma*Q!--MSo1u{L*^Jz^%k)UL@^K~#>cXc(T}$byz&Ed zl20Rm4LRgp53n}Qll#DR>b?(I2szGuRYF=x4U8CjGb{z8{w8wKvYawR02XT&c0@^I zf*oRM@d-6_M79QcqeG2SS;fXc8nlPZ6ev#p!1U4@zm97a4Ab-Y*@zcnB?3qW((cdldG2t0GCR7R?oCyj`oob_1skf_ct6VJB>cwWGSuMFPlGKxTC<&98 z9%7X-t5IkHrHq6sJ@f)CIZTu!vXQ_S6sVYa<;o@6_mn9cbhYs0qO;(snuiBnL((0OWU;u|sMjhDBh%4-V-;}4llcD&_r8;HQpVgZV=8Mv^gb96?cO#>EF_iN zrws}a(c+9vkeM7ythRE@t@qgd({j$?s>7({Fp zk>{Oh!k%#-e~%(OrI|8kGG|(njBPzL9L5p{Zx#yDUx}c!Yuax72`6rRP8p#(W_8w0 zJ44u@09gs}{zjxR2#ZA19j`^{`aV2C0dW3*3VN<=U;rBwpdT+^yIRgpk5@Yz6revn za)1pAV1okimU3dC0vi;-nX^FwNH=M0+EmOM8x#QUv*Q(qc0GE(oPR{ioE3ENG&;A_ zhF-ruksT9v2RTOo8)~W>dav1_034l@w?L$nw?P4X24wad5VCH7_H*|v9fNLz0w~w> z9BE}^L*905NH}m#xcp76L`ed~Tj~KDHFDqxI;8WVdN=pGHYh-J0`?7sg<=U_wy>~s zO<6H<{pLF{od1ta6h&Nu?XF<^5(m=N8-x6PMf$9)B3P<9%45JIB1m^+hGzoKHc6++m$W$25r6$ekb!3Ao zsdtrbi6mH7wyVuBZ5r%nbh02eMIu^`4FL2+ zPp>;Gpe5px*IrP%`>c3AKkC5;2$^I&*o5E*spTIbFt8#xoBBstJ zxTXuK$@Rn%FEP{8Ys`whS4N1Wk(G5{qsKJuhWG&`gLi+chh>~d$Q?yUca+!MGXJ87B)!55xkp2s{}05+!TfXUc#_ZZ z;U$ly({w<78RRgN!E@qm1hffM`}E8e`T&(34jS#b9L*m7WgMzpR~iSpq`qfgd_Qcc zGEzWODrPF4GBw9|l%9m$2nBY}U�Ay27seV*fbFu`9a?f~Pwh)xzD0W*cdG;Bp=f zN9+y-|4Xh^dHDAHHr!i9Zp7k8N77UkG=nG#dUst&OkAq=*ess$z>NXY5oQd?sK!qhO9@T&%H2Lr+#ewuBB zWO12Q`59g0S)9tf=xxj+N*|m2K6hGQQ)m|A@EPpjif9)`0utiE8IF4Cw7ay*WMq^u z3PDM=G9{;aI(1~IGi{&El`RCF1+^xI*^P@(1}1w&lwtX&G>G{Q90(<72G>gTp=(VX z-{L)Ti1-;2{D9n02UvpOZK!1x|URN37kQB$m=7*#I z-gPOaN0P@``e={^C4Or*2AL>y1kk*;FAKtoV@*<2@1SKcuHPH=@OqJmZlXZeb&Uye zM?3WNeYYGNXp+zKsXwo~7l{OGlUvk>Pf*7 zQj8A-Xc!=G5m-$2bmaL<*nb#PC91K;@;qPMBmM*JUV@c+KM)7=6A0{uy6=HJ5x;w17lcmjs^afYP9Yb6P#Ezg~N63U= zG?&k}`9~j3t$A2{C1`4;1jQ#2vfgb&#8o|}D5qxRV zHLr24t1Z9c)197~iSX-lm5%qEIa_PnuMg{CO%s(l@5b~H8OUYMbcvji=JBGCBN>Q} z4HlhV`_Cc|DWC~^_o@npLDFMe}V#B)6@D4jRTxj_4D=XJ$O@!HBW1RbMB1vmh|?l$M#s)WmWy% zi)N$|jxUyT4;F82a`#HcL{Hz?WDl}(thTCt8}UV-U&c$t2TC;rK1e8lcaVCT=BJsM zq)y{|EdGS8bGm6&{gW)@<#d>y>D`DuRu*P2@V;hM{jsCXoK^MXDlS*ePx}82Tsg_P zHLYdNV`;1E7ej@3r#uVOCjDec9jKz_SXKXr^H!e~ZC2Gk$Lj7JF(L|GtLpb&!ydpq z4dq}JUsG6BKNNvjtmc*}y0=yJ=Ni%I3lXd82WJ_2^&G~ss(xZ7)|p=JLhnwe#H1_& zr&(3M#0{0SBX@Kioukmpp-QXjPuyU!sx=2DtLkqfguYevd#{YlaJH<)5W56OGi|7F Z2Zyx}sHz_pHn8!5SquIff5_ literal 0 HcmV?d00001 diff --git a/opt/02-core/presenton/app_data/userConfig.json b/opt/02-core/presenton/app_data/userConfig.json new file mode 100644 index 0000000..980a4a0 --- /dev/null +++ b/opt/02-core/presenton/app_data/userConfig.json @@ -0,0 +1 @@ +{"LLM":"google","OPENAI_API_KEY":"4roCnNdmB9PpRuW0cGcU66bFBU6pUcUgeEsuP3a3lGmxwuZTWD3mJQQJ99BKACfhMk5XJ3w3AAAAACOGcIX4","OPENAI_MODEL":"gpt-5","GOOGLE_API_KEY":"AIzaSyC5Tsf57X9egANg_ft3aFA_59sTL8i8gwA","GOOGLE_MODEL":"models/gemini-2.5-flash","OLLAMA_URL":"http://localhost:11434","CUSTOM_LLM_URL":"http://litellm-proxy:4000","CUSTOM_LLM_API_KEY":"presenton-proxy-key-2025","CUSTOM_MODEL":"gpt-5","IMAGE_PROVIDER":"gemini_flash","TOOL_CALLS":"true","EXTENDED_REASONING":"true"} \ No newline at end of file diff --git a/opt/05-backups/scripts/backup-full-enhanced.sh b/opt/05-backups/scripts/backup-full-enhanced.sh new file mode 100755 index 0000000..96f99cf --- /dev/null +++ b/opt/05-backups/scripts/backup-full-enhanced.sh @@ -0,0 +1,467 @@ +#!/bin/bash + +################################################################################ +# AI-Impress Enhanced Full Backup System +# Version: 2.1.0 +# Purpose: Auto-discover and backup all system components +# Features: +# - Auto-discovery of docker-compose projects +# - Automatic database detection (PostgreSQL, MariaDB/MySQL, MongoDB) +# - Incremental backups with Restic +# - Local backup on /mnt/backups +# - Slack & Email notifications +# Author: AI-Impress Admin System +# Date: 2025-11-06 +# Changelog v2.1.0: +# - Added auto-discovery for all database types +# - Improved database detection with image inspection +# - Added MongoDB backup support +# - Better error handling +################################################################################ + +set -e + +# ============================================ +# CONFIGURATION +# ============================================ + +RESTIC_ENV="/opt/05-backups/restic/.env" +BACKUP_BASE="/mnt/backups" +LOCAL_BACKUP_DIR="$BACKUP_BASE/local-backups" +REPORTS_DIR="/opt/05-backups/reports" +LOG_DIR="/opt/05-backups/logs" +LOG_FILE="$LOG_DIR/backup-$(date +%Y%m%d-%H%M%S).log" +BACKUP_REPORT="$REPORTS_DIR/backup-report-$(date +%Y%m%d-%H%M%S).json" + +SLACK_WEBHOOK="${SLACK_WEBHOOK_URL:-}" +EMAIL_TO="admin@ai-impress.com" + +# Load Restic credentials +if [[ -f "$RESTIC_ENV" ]]; then + source "$RESTIC_ENV" +else + echo "ERROR: Restic .env not found at $RESTIC_ENV" + exit 1 +fi + +# Export Vault token +export VAULT_TOKEN="${VAULT_TOKEN:-$(cat /opt/00-infrastructure/vault/.vault-token 2>/dev/null)}" + +# Create directories +mkdir -p "$BACKUP_BASE" "$LOCAL_BACKUP_DIR" "$REPORTS_DIR" "$LOG_DIR" + +# Redirect output to log +exec 1> >(tee -a "$LOG_FILE") +exec 2>&1 + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +# ============================================ +# HELPER FUNCTIONS +# ============================================ + +log() { + echo "[$(date +%Y-%m-%d\ %H:%M:%S)] $1" +} + +success() { + echo -e "${GREEN}✅ $1${NC}" +} + +warning() { + echo -e "${YELLOW}⚠️ $1${NC}" +} + +error() { + echo -e "${RED}❌ $1${NC}" +} + +send_slack() { + local title=$1 + local message=$2 + local color=${3:-good} + + if [[ -z "$SLACK_WEBHOOK" ]]; then + return + fi + + curl -X POST "$SLACK_WEBHOOK" \ + -H 'Content-Type: application/json' \ + -d "{ + \"attachments\": [{ + \"color\": \"$color\", + \"title\": \"$title\", + \"text\": \"$message\", + \"footer\": \"AI-Impress Backup System\", + \"ts\": $(date +%s) + }] + }" 2>/dev/null || true +} + +send_email() { + local subject=$1 + local body=$2 + echo "$body" | mail -s "$subject" "$EMAIL_TO" 2>/dev/null || true +} + +# Initialize backup report +init_report() { + cat > "$BACKUP_REPORT" << EOF +{ + "timestamp": "$(date -u +%Y-%m-%dT%H:%M:%SZ)", + "hostname": "$(hostname)", + "backup_status": "IN_PROGRESS", + "components": {}, + "summary": { + "total_components": 0, + "successful": 0, + "failed": 0, + "total_size": "0GB", + "duration": "calculating" + }, + "alerts": [] +} +EOF +} + +################################################################################ +# AUTO-DISCOVERY FUNCTIONS +################################################################################ + +discover_docker_compose_projects() { + log "=== Auto-discovering Docker Compose Projects ===" + + local projects=() + + # Scan /opt for docker-compose.yml files + while IFS= read -r compose_file; do + local project_path=$(dirname "$compose_file") + local project_name=$(basename "$project_path") + projects+=("$project_path") + log "Found: $project_path" + done < <(find /opt -maxdepth 4 -name "docker-compose.yml" -type f 2>/dev/null) + + echo "${projects[@]}" +} + +discover_databases() { + log "=== Auto-discovering Databases ===" + + local databases=() + + # Auto-detect PostgreSQL containers (common image names and postgres in container name) + while IFS= read -r container; do + if docker inspect "$container" --format '{{.Config.Image}}' 2>/dev/null | grep -qiE '(postgres|postgresql|timescale|postgis)'; then + databases+=("postgresql:$container") + log "Found PostgreSQL: $container" + fi + done < <(docker ps --format '{{.Names}}' 2>/dev/null | grep -iE '(postgres|pg|timescale|supabase-db|authentik-postgres|postiz-postgres)') + + # Auto-detect MariaDB/MySQL containers + while IFS= read -r container; do + if docker inspect "$container" --format '{{.Config.Image}}' 2>/dev/null | grep -qiE '(mariadb|mysql)'; then + databases+=("mariadb:$container") + log "Found MariaDB/MySQL: $container" + fi + done < <(docker ps --format '{{.Names}}' 2>/dev/null | grep -iE '(mariadb|mysql|mautic-db)') + + # Auto-detect MongoDB containers + while IFS= read -r container; do + if docker inspect "$container" --format '{{.Config.Image}}' 2>/dev/null | grep -qiE 'mongo'; then + databases+=("mongodb:$container") + log "Found MongoDB: $container" + fi + done < <(docker ps --format '{{.Names}}' 2>/dev/null | grep -iE 'mongo') + + echo "${databases[@]}" +} + +discover_volumes() { + log "=== Auto-discovering Docker Volumes ===" + + local volumes=() + + # Get all used volumes + while IFS= read -r volume; do + volumes+=("$volume") + log "Found volume: $volume" + done < <(docker volume ls --format "{{.Name}}" 2>/dev/null | grep -v "^$") + + echo "${volumes[@]}" +} + +################################################################################ +# BACKUP FUNCTIONS +################################################################################ + +backup_postgresql() { + local container=$1 + local db_user=${2:-aimpress_admin} + local backup_file="$LOCAL_BACKUP_DIR/postgresql-$container-$(date +%Y%m%d-%H%M%S).sql.gz" + + log "[DB] Backing up PostgreSQL: $container" + + if ! docker ps --filter "name=$container" -q &>/dev/null; then + warning "PostgreSQL container $container not running" + return 1 + fi + + if docker exec "$container" pg_dumpall -U $db_user 2>/dev/null | gzip > "$backup_file"; then + local size=$(du -h "$backup_file" | cut -f1) + success "PostgreSQL $container backed up ($size)" + + # Keep only last 14 days + find "$LOCAL_BACKUP_DIR" -name "postgresql-$container-*.sql.gz" -mtime +14 -delete + + return 0 + else + error "Failed to backup PostgreSQL: $container" + return 1 + fi +} + +backup_mariadb() { + local container=$1 + local backup_file="$LOCAL_BACKUP_DIR/mariadb-$container-$(date +%Y%m%d-%H%M%S).sql.gz" + + log "[DB] Backing up MariaDB: $container" + + if ! docker ps --filter "name=$container" -q &>/dev/null; then + warning "MariaDB container $container not running" + return 1 + fi + + if docker exec "$container" mariadb-dump --all-databases 2>/dev/null | gzip > "$backup_file"; then + local size=$(du -h "$backup_file" | cut -f1) + success "MariaDB $container backed up ($size)" + + # Keep only last 14 days + find "$LOCAL_BACKUP_DIR" -name "mariadb-$container-*.sql.gz" -mtime +14 -delete + + return 0 + else + error "Failed to backup MariaDB: $container" + return 1 + fi +} + +backup_mongodb() { + local container=$1 + local backup_file="$LOCAL_BACKUP_DIR/mongodb-$container-$(date +%Y%m%d-%H%M%S).gz" + + log "[DB] Backing up MongoDB: $container" + + if ! docker ps --filter "name=$container" -q &>/dev/null; then + warning "MongoDB container $container not running" + return 1 + fi + + if docker exec "$container" mongodump --archive 2>/dev/null | gzip > "$backup_file"; then + local size=$(du -h "$backup_file" | cut -f1) + success "MongoDB $container backed up ($size)" + + # Keep only last 14 days + find "$LOCAL_BACKUP_DIR" -name "mongodb-$container-*.gz" -mtime +14 -delete + + return 0 + else + error "Failed to backup MongoDB: $container" + return 1 + fi +} + +backup_vault() { + log "[CONFIG] Backing up Vault..." + local backup_file="$LOCAL_BACKUP_DIR/vault-data-$(date +%Y%m%d-%H%M%S).tar.gz" + + if sudo tar czf "$backup_file" -C /opt/00-infrastructure/vault data 2>/dev/null; then + local size=$(du -h "$backup_file" | cut -f1) + success "Vault data backed up ($size)" + + # Keep only last 30 days + find "$LOCAL_BACKUP_DIR" -name "vault-data-*.tar.gz" -mtime +30 -delete + + return 0 + else + error "Failed to backup Vault" + return 1 + fi +} + +backup_docker_configs() { + log "[CONFIG] Backing up Docker Compose files..." + local backup_file="$LOCAL_BACKUP_DIR/docker-configs-$(date +%Y%m%d-%H%M%S).tar.gz" + + if tar czf "$backup_file" -C /opt . -path "*docker-compose.yml" 2>/dev/null; then + local size=$(du -h "$backup_file" | cut -f1) + success "Docker configs backed up ($size)" + + # Keep only last 30 days + find "$LOCAL_BACKUP_DIR" -name "docker-configs-*.tar.gz" -mtime +30 -delete + + return 0 + else + error "Failed to backup Docker configs" + return 1 + fi +} + +backup_application_data() { + log "[DATA] Backing up Application Data..." + local backup_file="$LOCAL_BACKUP_DIR/app-data-$(date +%Y%m%d-%H%M%S).tar.gz" + + local data_dirs=( + "/opt/03-business/mautic/sync_v2" + "/opt/02-core/supabase/supabase/docker/volumes" + ) + + if tar czf "$backup_file" "${data_dirs[@]}" 2>/dev/null; then + local size=$(du -h "$backup_file" | cut -f1) + success "Application data backed up ($size)" + + # Keep only last 14 days + find "$LOCAL_BACKUP_DIR" -name "app-data-*.tar.gz" -mtime +14 -delete + + return 0 + else + error "Failed to backup application data" + return 1 + fi +} + +backup_with_restic() { + log "=== Uploading to Restic (Cloudflare R2) ===" + + if ! command -v restic &>/dev/null; then + warning "Restic not installed, skipping cloud backup" + return 1 + fi + + # Initialize Restic repository if needed + if ! restic cat config &>/dev/null; then + log "Initializing Restic repository..." + restic init || warning "Restic repository might already exist" + fi + + # Backup local directory to Restic + if restic backup "$BACKUP_BASE" --exclude-file="$BACKUP_BASE/.restic-exclude" 2>/dev/null; then + success "Restic backup completed" + + # Cleanup old snapshots (keep last 30) + restic forget --keep-daily 3 --keep-weekly 1 --prune 2>/dev/null || true + + return 0 + else + error "Restic backup failed" + return 1 + fi +} + +################################################################################ +# MAIN BACKUP EXECUTION +################################################################################ + +main() { + local start_time=$(date +%s) + + log "╔════════════════════════════════════════════════════════════╗" + log "║ AI-Impress Enhanced Full Backup v2.1.0 ║" + log "║ $(date +%Y-%m-%d\ %H:%M:%S) ║" + log "╚════════════════════════════════════════════════════════════╝" + log "" + + local failed=0 + local successful=0 + + # Auto-discover and backup databases + log "=== PHASE 1: Database Backups (Auto-Discovery) ===" + + # Discover all databases automatically + local discovered_dbs=$(discover_databases) + + if [[ -z "$discovered_dbs" ]]; then + warning "No databases discovered" + else + log "Found databases: $discovered_dbs" + log "" + + for db in $discovered_dbs; do + local db_type=$(echo "$db" | cut -d: -f1) + local db_container=$(echo "$db" | cut -d: -f2) + + case $db_type in + postgresql) + # Determine DB user based on container name + local db_user="aimpress_admin" + [[ "$db_container" == "authentik-postgres" ]] && db_user="authentik" + [[ "$db_container" == "postiz-postgres" ]] && db_user="postiz" + + backup_postgresql "$db_container" "$db_user" && ((successful++)) || ((failed++)) + ;; + mariadb) + backup_mariadb "$db_container" && ((successful++)) || ((failed++)) + ;; + mongodb) + backup_mongodb "$db_container" && ((successful++)) || ((failed++)) + ;; + *) + warning "Unknown database type: $db_type" + ((failed++)) + ;; + esac + done + fi + + log "" + log "=== PHASE 2: Configuration Backups ===" + + backup_vault && ((successful++)) || ((failed++)) + backup_docker_configs && ((successful++)) || ((failed++)) + + log "" + log "=== PHASE 3: Application Data ===" + + backup_application_data && ((successful++)) || ((failed++)) + + log "" + log "=== PHASE 4: Cloud Backup (Restic) ===" + + backup_with_restic && ((successful++)) || ((failed++)) + + # Calculate duration + local end_time=$(date +%s) + local duration=$((end_time - start_time)) + local duration_min=$((duration / 60)) + local duration_sec=$((duration % 60)) + + log "" + log "╔════════════════════════════════════════════════════════════╗" + log "║ BACKUP COMPLETE ║" + log "╚════════════════════════════════════════════════════════════╝" + log "" + log "Summary:" + log " Total Components: $((successful + failed))" + log " Successful: $successful" + log " Failed: $failed" + log " Duration: ${duration_min}m ${duration_sec}s" + log " Local Backups: $LOCAL_BACKUP_DIR" + log " Cloud Backups: Restic (Cloudflare R2)" + log "" + + # Send notifications + if [[ $failed -eq 0 ]]; then + success "All backups completed successfully!" + send_slack "✅ Backup Complete" "All components backed up successfully in ${duration_min}m ${duration_sec}s" "good" + send_email "Backup Complete" "All backups completed successfully.\n\nDuration: ${duration_min}m ${duration_sec}s\nLocation: $BACKUP_BASE" + else + warning "Backup completed with $failed failures" + send_slack "⚠️ Backup Completed with Errors" "Failed components: $failed\nCheck logs: $LOG_FILE" "warning" + send_email "Backup Completed with Errors" "Backup completed with $failed failures.\n\nCheck logs: $LOG_FILE" + fi +} + +main "$@" diff --git a/opt/05-backups/scripts/health-check-alerting.sh b/opt/05-backups/scripts/health-check-alerting.sh new file mode 100755 index 0000000..5613976 --- /dev/null +++ b/opt/05-backups/scripts/health-check-alerting.sh @@ -0,0 +1,377 @@ +#!/bin/bash + +################################################################################ +# AI-Impress Health Check & Alerting System +# Version: 1.0.0 +# Purpose: Monitor system health and send alerts on problems +################################################################################ + +set -euo pipefail + +# Configuration +# Get from Vault +export VAULT_ADDR=http://127.0.0.1:8200 +export VAULT_TOKEN=$(cat /opt/00-infrastructure/vault/.vault-token 2>/dev/null || echo "") +SLACK_WEBHOOK_URL=$(vault kv get -field=slack_webhook secret/monitoring 2>/dev/null || echo "") # Set in Vault or environment +ALERT_EMAIL="${ALERT_EMAIL:-admin@ai-impress.com}" +SMTP_SERVER="${SMTP_SERVER:-localhost}" +LOG_FILE="/opt/05-backups/logs/health-check-$(date +%Y%m%d).log" + +# Thresholds +DISK_THRESHOLD=90 # Alert if disk > 90% +MEMORY_THRESHOLD=90 # Alert if memory > 90% +MAX_UNHEALTHY_CONTAINERS=2 # Alert if more than 2 containers unhealthy + +# Colors +RED='\033[0;31m' +GREEN='\033[0;32m' +YELLOW='\033[1;33m' +NC='\033[0m' + +PROBLEMS=() +WARNINGS=() + +log() { + echo "[$(date '+%Y-%m-%d %H:%M:%S')] $1" | tee -a "$LOG_FILE" +} + +add_problem() { + PROBLEMS+=("$1") + log "🔴 PROBLEM: $1" +} + +add_warning() { + WARNINGS+=("$1") + log "🟡 WARNING: $1" +} + +################################################################################ +# CHECKS +################################################################################ + +check_critical_services() { + log "Checking critical services..." + + local critical_services=("traefik" "postgres-main" "redis-main") + + for service in "${critical_services[@]}"; do + if ! docker ps --format '{{.Names}}' | grep -q "^${service}$"; then + add_problem "Critical service $service is NOT RUNNING" + else + local health=$(docker inspect --format='{{.State.Health.Status}}' "$service" 2>/dev/null || echo "no healthcheck") + if [[ "$health" == "unhealthy" ]]; then + add_problem "Critical service $service is UNHEALTHY" + fi + fi + done +} + +check_websites() { + log "Checking websites..." + + local websites=( + "wiki.ai-impress.com" + "n8n.ai-impress.com" + "odoo.ai-impress.com" + "auth.ai-impress.com" + ) + + for site in "${websites[@]}"; do + local http_code=$(curl -s -o /dev/null -w "%{http_code}" --max-time 10 "https://$site" 2>/dev/null || echo "000") + + if [[ "$http_code" != "200" ]] && [[ ! "$http_code" =~ ^30 ]]; then + add_problem "Website $site is DOWN (HTTP $http_code)" + fi + done +} + +check_disk_space() { + log "Checking disk space..." + + local disk_usage=$(df -h / | awk 'NR==2 {print $5}' | sed 's/%//') + + if [[ "$disk_usage" -gt "$DISK_THRESHOLD" ]]; then + add_problem "Disk usage is CRITICAL: ${disk_usage}% (threshold: ${DISK_THRESHOLD}%)" + elif [[ "$disk_usage" -gt 80 ]]; then + add_warning "Disk usage is high: ${disk_usage}%" + fi + + # Check /mnt/psql-data + if [[ -d /mnt/psql-data ]]; then + local db_disk_usage=$(df -h /mnt/psql-data | awk 'NR==2 {print $5}' | sed 's/%//') + if [[ "$db_disk_usage" -gt "$DISK_THRESHOLD" ]]; then + add_problem "Database disk usage is CRITICAL: ${db_disk_usage}%" + fi + fi +} + +check_memory() { + log "Checking memory usage..." + + local memory_usage=$(free | awk '/Mem:/ {printf "%.0f", ($3/$2)*100}') + + if [[ "$memory_usage" -gt "$MEMORY_THRESHOLD" ]]; then + add_problem "Memory usage is CRITICAL: ${memory_usage}% (threshold: ${MEMORY_THRESHOLD}%)" + elif [[ "$memory_usage" -gt 80 ]]; then + add_warning "Memory usage is high: ${memory_usage}%" + fi +} + +check_unhealthy_containers() { + log "Checking unhealthy containers..." + + local unhealthy_count=$(docker ps --filter "health=unhealthy" -q | wc -l) + + if [[ "$unhealthy_count" -gt "$MAX_UNHEALTHY_CONTAINERS" ]]; then + local containers=$(docker ps --filter "health=unhealthy" --format '{{.Names}}' | tr '\n' ', ') + add_problem "$unhealthy_count containers are UNHEALTHY: $containers" + elif [[ "$unhealthy_count" -gt 0 ]]; then + local containers=$(docker ps --filter "health=unhealthy" --format '{{.Names}}' | tr '\n' ', ') + add_warning "$unhealthy_count container(s) unhealthy: $containers" + fi +} + +check_r2_usage() { + log "Checking Cloudflare R2 backup storage..." + + # Load Restic environment + if [[ ! -f /opt/05-backups/restic/.env ]]; then + add_warning "Restic config not found - skipping R2 check" + return + fi + + source /opt/05-backups/restic/.env + + # Get R2 stats + local r2_stats=$(restic stats --mode restore-size 2>/dev/null | grep "Total Size") + + if [[ -z "$r2_stats" ]]; then + add_warning "Unable to get R2 statistics" + return + fi + + local size_gb=$(echo "$r2_stats" | grep -oP '\d+\.\d+' | head -1) + local r2_limit=10 # Cloudflare R2 free tier limit: 10 GB + + # Check if bc is available for floating point comparison + if command -v bc &> /dev/null; then + if (( $(echo "$size_gb > $r2_limit" | bc -l) )); then + add_problem "R2 storage EXCEEDED: ${size_gb}GB / ${r2_limit}GB limit" + elif (( $(echo "$size_gb > 8" | bc -l) )); then + add_warning "R2 storage high: ${size_gb}GB / ${r2_limit}GB (>80%)" + else + log "✅ R2 storage OK: ${size_gb}GB / ${r2_limit}GB" + fi + else + # Fallback: use integer comparison if bc not available + local size_gb_int=$(echo "$size_gb" | cut -d. -f1) + if [[ "$size_gb_int" -gt "$r2_limit" ]]; then + add_problem "R2 storage EXCEEDED: ${size_gb}GB / ${r2_limit}GB limit" + elif [[ "$size_gb_int" -gt 8 ]]; then + add_warning "R2 storage high: ${size_gb}GB / ${r2_limit}GB (>80%)" + else + log "✅ R2 storage OK: ${size_gb}GB / ${r2_limit}GB" + fi + fi + + # Check snapshot count + local snapshot_count=$(restic snapshots --compact 2>/dev/null | grep -c "^[a-f0-9]" || echo "0") + log "📦 R2 snapshots: $snapshot_count (policy: keep 3 daily + 1 weekly)" + + if [[ $snapshot_count -gt 5 ]]; then + add_warning "Too many R2 snapshots: $snapshot_count (expected ≤4)" + fi +} + +check_backup_status() { + log "Checking backup status..." + + if [[ ! -d /mnt/backups ]]; then + add_problem "Backup directory /mnt/backups NOT FOUND" + return + fi + + local latest_backup=$(find /mnt/backups -type f -name "*.tar.gz" -o -name "*.sql.gz" 2>/dev/null | sort | tail -1) + + if [[ -z "$latest_backup" ]]; then + add_problem "NO BACKUPS FOUND in /mnt/backups" + else + local backup_age_days=$(( ($(date +%s) - $(stat -c %Y "$latest_backup")) / 86400 )) + + if [[ "$backup_age_days" -gt 2 ]]; then + add_problem "Latest backup is $backup_age_days days old (last: $(basename "$latest_backup"))" + elif [[ "$backup_age_days" -gt 1 ]]; then + add_warning "Latest backup is $backup_age_days days old" + fi + fi +} + +check_container_restarts() { + log "Checking for excessive container restarts..." + + while read -r container restart_count; do + if [[ "$restart_count" -gt 10 ]]; then + add_problem "Container $container has restarted $restart_count times" + elif [[ "$restart_count" -gt 5 ]]; then + add_warning "Container $container has restarted $restart_count times" + fi + done < <(docker ps --format '{{.Names}}' | xargs -I {} sh -c 'echo {} $(docker inspect --format="{{.RestartCount}}" {})') +} + +################################################################################ +# ALERT SENDING +################################################################################ + +send_slack_alert() { + local message="$1" + + if [[ -z "$SLACK_WEBHOOK_URL" ]]; then + log "Slack webhook not configured, skipping..." + return + fi + + local payload=$(cat </dev/null || log "Failed to send Slack alert" +} + +send_email_alert() { + local subject="$1" + local body="$2" + + # Try to send via mail command if available + if command -v mail &> /dev/null; then + echo "$body" | mail -s "$subject" "$ALERT_EMAIL" 2>/dev/null || log "Failed to send email" + elif command -v sendmail &> /dev/null; then + echo -e "Subject: $subject\nTo: $ALERT_EMAIL\n\n$body" | sendmail "$ALERT_EMAIL" 2>/dev/null || log "Failed to send email" + else + log "No mail command available, saving email to file" + echo -e "To: $ALERT_EMAIL\nSubject: $subject\n\n$body" > "/tmp/alert-email-$(date +%s).txt" + fi +} + +generate_alert_report() { + local report="🚨 AI-Impress Server Health Alert\n\n" + report+="Server: ai-impress-prod (51.89.231.46)\n" + report+="Time: $(date '+%Y-%m-%d %H:%M:%S')\n\n" + + if [[ ${#PROBLEMS[@]} -gt 0 ]]; then + report+="🔴 CRITICAL PROBLEMS (${#PROBLEMS[@]}):\n" + for problem in "${PROBLEMS[@]}"; do + report+=" - $problem\n" + done + report+="\n" + fi + + if [[ ${#WARNINGS[@]} -gt 0 ]]; then + report+="🟡 WARNINGS (${#WARNINGS[@]}):\n" + for warning in "${WARNINGS[@]}"; do + report+=" - $warning\n" + done + report+="\n" + fi + + report+="━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n" + report+="🔧 HOW TO FIX:\n\n" + report+="1. SSH to server:\n" + report+=" ssh ubuntu@51.89.231.46\n\n" + report+="2. Check full status:\n" + report+=" /opt/05-backups/scripts/admin.sh status\n\n" + report+="3. View detailed logs:\n" + report+=" docker logs --tail 100\n\n" + report+="4. Restart service if needed:\n" + report+=" docker restart \n\n" + report+="5. Check disk space:\n" + report+=" df -h\n\n" + report+="━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n\n" + report+="📊 Quick System Status:\n" + report+=" Memory: $(free -h | awk '/^Mem:/ {print $3 "/" $2}')\n" + report+=" Disk: $(df -h / | awk 'NR==2 {print $3 "/" $2 " (" $5 ")"}')\n" + report+=" Containers: $(docker ps -q | wc -l) running\n" + report+=" Uptime: $(uptime -p)\n\n" + report+="━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n" + report+="Generated by: /opt/05-backups/scripts/health-check-alerting.sh\n" + + echo -e "$report" +} + +################################################################################ +# MAIN +################################################################################ + +main() { + log "╔══════════════════════════════════════════════════════╗" + log "║ AI-Impress Health Check & Alerting System ║" + log "║ $(date '+%Y-%m-%d %H:%M:%S') ║" + log "╚══════════════════════════════════════════════════════╝" + log "" + + # Run all checks + check_critical_services + check_websites + check_disk_space + check_memory + check_r2_usage + check_unhealthy_containers + check_backup_status + check_container_restarts + + log "" + log "Summary: ${#PROBLEMS[@]} problems, ${#WARNINGS[@]} warnings" + + # Send alerts if there are problems + if [[ ${#PROBLEMS[@]} -gt 0 ]]; then + log "🚨 CRITICAL PROBLEMS DETECTED - Sending alerts..." + + local alert_report=$(generate_alert_report) + + # Send to Slack + send_slack_alert "$alert_report" + + # Send via Email + send_email_alert "🚨 AI-Impress Server Alert - ${#PROBLEMS[@]} Critical Problems" "$alert_report" + + echo -e "\n${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo -e "${RED}⚠️ CRITICAL PROBLEMS DETECTED!${NC}" + echo -e "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + for problem in "${PROBLEMS[@]}"; do + echo -e "${RED} • $problem${NC}" + done + echo -e "${RED}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}\n" + + exit 1 + elif [[ ${#WARNINGS[@]} -gt 0 ]]; then + log "⚠️ Warnings detected (no critical problems)" + + echo -e "\n${YELLOW}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo -e "${YELLOW}⚠️ WARNINGS DETECTED${NC}" + echo -e "${YELLOW}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + for warning in "${WARNINGS[@]}"; do + echo -e "${YELLOW} • $warning${NC}" + done + echo -e "${YELLOW}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}\n" + + exit 0 + else + log "✅ All checks passed - System is healthy" + echo -e "\n${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}" + echo -e "${GREEN}✅ All checks passed - System is healthy${NC}" + echo -e "${GREEN}━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━${NC}\n" + + exit 0 + fi +} + +# Run if executed directly +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main "$@" +fi diff --git a/opt/infrastructure-docs/scripts/modules/generate-databases.sh b/opt/infrastructure-docs/scripts/modules/generate-databases.sh index 6556c5f..31ba41d 100755 --- a/opt/infrastructure-docs/scripts/modules/generate-databases.sh +++ b/opt/infrastructure-docs/scripts/modules/generate-databases.sh @@ -108,7 +108,7 @@ docker exec postgres-main psql -U aimpress_admin -c \ # 3. Save to Vault export VAULT_ADDR="http://127.0.0.1:8200" -export VAULT_TOKEN="hvs.jYguDdf2IzobXG8b9QWyATV8" +export VAULT_TOKEN=$(cat /opt/00-infrastructure/vault/.vault-token) vault kv put aimpress/postgres/ password="$NEW_PASS" # 4. Update application config diff --git a/opt/infrastructure-docs/scripts/modules/generate-summary.sh b/opt/infrastructure-docs/scripts/modules/generate-summary.sh index f1a839c..237b30f 100755 --- a/opt/infrastructure-docs/scripts/modules/generate-summary.sh +++ b/opt/infrastructure-docs/scripts/modules/generate-summary.sh @@ -64,5 +64,6 @@ for service in traefik postgres-main redis-main vault; do fi done -cat << 'EOF' - +echo "" +echo "---" +echo "" diff --git a/opt/infrastructure-docs/scripts/modules/generate-websites.sh b/opt/infrastructure-docs/scripts/modules/generate-websites.sh index d8fc397..1f1f4af 100755 --- a/opt/infrastructure-docs/scripts/modules/generate-websites.sh +++ b/opt/infrastructure-docs/scripts/modules/generate-websites.sh @@ -105,7 +105,7 @@ ssh ubuntu@51.89.231.46 # Set Vault variables export VAULT_ADDR="http://127.0.0.1:8200" -export VAULT_TOKEN="hvs.jYguDdf2IzobXG8b9QWyATV8" +export VAULT_TOKEN=$(cat /opt/00-infrastructure/vault/.vault-token) # List all available secrets vault kv list aimpress/ diff --git a/opt/infrastructure-docs/scripts/server-full-report.sh b/opt/infrastructure-docs/scripts/server-full-report.sh index 84398d0..50374d2 100755 --- a/opt/infrastructure-docs/scripts/server-full-report.sh +++ b/opt/infrastructure-docs/scripts/server-full-report.sh @@ -26,6 +26,126 @@ log() { echo -e "${CYAN}[$(date +%H:%M:%S)]${NC} $1"; } success() { echo -e "${GREEN}✅ $1${NC}"; } error() { echo -e "${RED}❌ $1${NC}"; exit 1; } +# Slack notification function +send_slack_summary() { + local report_file="$1" + + # Get Vault token and Slack webhook + export VAULT_ADDR="http://127.0.0.1:8200" + export VAULT_TOKEN=$(cat /opt/00-infrastructure/vault/.vault-token 2>/dev/null || echo "") + local slack_webhook=$(vault kv get -field=slack_webhook secret/monitoring 2>/dev/null || echo "") + + if [[ -z "$slack_webhook" ]]; then + log "Slack webhook not configured, skipping notification" + return 0 + fi + + # Extract key metrics from report + local containers_running=$(grep "Docker Containers" "$report_file" | grep -oP '\d+(?= running)' | head -1) + local containers_total=$(grep "Docker Containers" "$report_file" | grep -oP 'running / \d+' | grep -oP '\d+' | head -1) + local unhealthy=$(grep "Unhealthy Containers" "$report_file" | grep -oP '\d+' | head -1) + local memory=$(grep "Memory" "$report_file" | grep -oP '\| \*\*Memory\*\* \| \K[^|]+' | xargs) + local disk=$(grep "Disk (/)" "$report_file" | grep -oP '\| \*\*Disk \(/\)\*\* \| \K[^|]+' | xargs) + local disk_percent=$(echo "$disk" | grep -oP '\d+(?=\%)') + local uptime=$(grep "Uptime" "$report_file" | grep -oP '\| \*\*Uptime\*\* \| \K[^|]+' | xargs) + + # Count websites status + local websites_ok=$(grep -c "✅ OK" "$report_file" 2>/dev/null || echo "0") + local websites_down=$(grep -c "❌" "$report_file" 2>/dev/null || echo "0") + + # Detect problems and create recommendations + local problems="" + local recommendations="" + local color="good" + local status_emoji="✅" + + if [[ "$unhealthy" -gt 0 ]]; then + problems="${problems}• $unhealthy unhealthy container(s) detected\n" + recommendations="${recommendations}• Check logs: \`docker logs \`\n• Restart if needed: \`docker restart \`\n" + color="danger" + status_emoji="🚨" + fi + + if [[ "$websites_down" -gt 0 ]]; then + problems="${problems}• $websites_down website(s) are down\n" + recommendations="${recommendations}• Check Traefik: \`docker logs traefik --tail 50\`\n• Verify DNS: \`nslookup \`\n• Check SSL certs: \`/opt/05-backups/scripts/admin.sh status websites\`\n" + if [[ "$color" != "danger" ]]; then + color="warning" + status_emoji="⚠️" + fi + fi + + if [[ -n "$disk_percent" ]] && [[ "$disk_percent" -gt 80 ]]; then + problems="${problems}• Disk usage is high: ${disk_percent}%\n" + recommendations="${recommendations}• Clean up old logs: \`/opt/05-backups/scripts/admin.sh cleanup logs\`\n• Clean up Docker: \`/opt/05-backups/scripts/admin.sh cleanup docker\`\n• Check disk: \`/opt/05-backups/scripts/admin.sh status disk\`\n" + if [[ "$color" == "good" ]]; then + color="warning" + status_emoji="⚠️" + fi + fi + + # Create fields array + local fields='[ + { + "title": "System Status", + "value": "🐳 Containers: '"$containers_running/$containers_total"' running\n💾 Memory: '"$memory"'\n💿 Disk: '"$disk"'\n⏱️ Uptime: '"$uptime"'", + "short": true + }, + { + "title": "Health Check", + "value": "🔴 Unhealthy: '"$unhealthy"' containers\n🌐 Websites: '"$websites_ok"' OK, '"$websites_down"' Down", + "short": true + }' + + # Add problems section if any + if [[ -n "$problems" ]]; then + fields+=', + { + "title": "⚠️ Detected Problems", + "value": "'"${problems}"'", + "short": false + }, + { + "title": "🔧 Recommended Actions", + "value": "'"${recommendations}"'SSH: \`ssh ubuntu@51.89.231.46\`\nAdmin tool: \`/opt/05-backups/scripts/admin.sh help\`", + "short": false + }' + fi + + fields+=', + { + "title": "Full Report", + "value": "📄 Generated: \`'"$(basename $report_file)"'\`\n📍 Location: \`/opt/infrastructure-docs/reports/\`\n📤 Upload to Wiki: \`/opt/05-backups/scripts/upload-to-outline.sh latest-report\`", + "short": false + } + ]' + + # Create Slack message + local payload='{ + "attachments": [ + { + "color": "'"$color"'", + "title": "'"$status_emoji"' Daily Server Report - '"$(date '+%Y-%m-%d %H:%M')"'", + "fields": '"$fields"', + "footer": "AI-Impress Infrastructure Monitor", + "footer_icon": "https://wiki.ai-impress.com/favicon.png", + "ts": '"$(date +%s)"' + } + ] + }' + + # Send to Slack + local response=$(curl -s -X POST "$slack_webhook" \ + -H 'Content-Type: application/json' \ + -d "$payload") + + if [[ "$response" == "ok" ]]; then + success "Slack summary sent with $(echo -e "$problems" | grep -c "•" || echo "0") problems detected" + else + log "Slack notification may have failed: $response" + fi +} + log "╔════════════════════════════════════════════════════════════╗" log "║ AI-Impress Complete Server Report Generator v5.0 ║" log "║ Modular Architecture - Full System Report ║" @@ -174,4 +294,9 @@ cat << EOFSTATS EOFSTATS +# Send Slack summary +log "" +log "Sending Slack summary..." +send_slack_summary "$REPORT_FILE" + exit 0