Monitor Create Projection on Vertica

We need to status of progress, when creating big tables/projections . After a bit googling I cannot see any monitoring scripts. Vertica documantation mention in about monitoring but It is not sufficient. So I typed my own script with trial and error. You can use this scripts to monitor progress of creation projection. Plesae inform me if you discover a problem with this scripts. In my cases this scripts run well.

Case1 : create a new superprojection or projection from an existing table. in my tracing I get 2 long running phases, so you will see 2 rows result, each row corresponds one phase. first step 1 reaches %100 and then step2 starts. After step 2 reaches %100, create projection will finish with success.

eg:

create projection public.monitor_projection_del_p
as select * from public.monitor_projection_del order by date, privateAddress,region;
select start_refresh();
(SELECT  'step 1' as 'step',
		round(((eep.counter_value)/ <anchor_table_row_nums>)*100,2)  as progress
FROM execution_engine_profiles  eep, projection_refreshes qp
where operator_name='StorageUnion' 	
	--and operator_name='ValExpr' 	--and operator_name='Sort' 	--and operator_name='TopK' 	--use anyone of this operator_name, result should be same
	and counter_name='rows produced'  
	and eep.is_executing
	and counter_value != 0
	and eep.transaction_id=qp.transaction_id
	and eep.session_id=	qp.session_id
	and qp.is_executing
	limit 1) --projection_refreshes tables returns 2 rows because of k-safety=1 . one of them is <projection_name_b0>  and other one is should be <projection_name_b1> . they are body projections and comlately have the same rows therefore have same storage on disk. with using limit 1 I get only one projction's information. It doesn't matter which one, both are same on all conditions. 
union all
select 'step 2' as 'step',  
		round(sum(progress)/2,2) as progress -- projection_refreshes tables returns 2 rows because of k-safety=1 . and sum(progress) reaches tottaly 200. so I will devide 2 to get "%100" return value from script.
from
(
SELECT  eep.counter_value/<anchor_table_row_nums>*100  as progress
FROM execution_engine_profiles  eep, projection_refreshes qp
where operator_name='DataTarget'
	and counter_name='written rows' 
	and counter_tag like '%isDV=f%'
	and eep.is_executing
	and counter_value != 0
	and eep.transaction_id=qp.transaction_id
	and eep.session_id=	qp.session_id
	and qp.is_executing
) a;

Case 2: create table as select (CTAS) statement. in my tracing I get 3 long running phases, so you will see three rows result, each row corresponds one phase. first step 1 reaches %100 and then step2 starts. After step 2 reaches %100, step 3 starts . After step 3 reaches %100, create projection will finish with success.

eg: Örnek:
create table public.monitor_projection_test as select * from (select * from public.netflow limit 500000000) a order by date,message;
SELECT  'step 1' as 'step',
         round(((eep.counter_value)/<anchor_table_row_nums>)*100,2) as 'progress %'
FROM execution_engine_profiles  eep, QUERY_PROFILES qp
where eep.session_id='gdatalab_db_node0003-212021:0x3ba7c4' --“create projection sesson”s id
       and operator_name='StorageUnion'
       and counter_name='rows produced'
       and  eep.is_executing
       and counter_value != 0
       and eep.transaction_id=qp.transaction_id
       and eep.session_id= qp.session_id
       and qp.is_executing
union all
SELECT   'step 2' as 'step',
         round(((eep.counter_value)/ <anchor_table_row_nums>)*100,2) as 'progress %'
FROM execution_engine_profiles  eep, QUERY_PROFILES qp
where eep.session_id='gdatalab_db_node0003-212021:0x3ba7c4' --“create projection sesson”s id
       and operator_name='ValExpr'  --and operator_name='Sort' --and operator_name='TopK' --use anyone of this operator_name, result should be same
       and counter_name='rows produced' 
       and eep.is_executing
       and counter_value != 0
       and eep.transaction_id=qp.transaction_id
       and eep.session_id= qp.session_id
       and qp.is_executing
union all
select  'step 3' as 'step',
		round(sum(progress),2) as 'progress %'
from
(
SELECT  
        eep.counter_value/<anchor_table_row_nums>*100 as 'progress'
FROM execution_engine_profiles  eep, QUERY_PROFILES qp
where eep.session_id='gdatalab_db_node0003-212021:0x3ba7c4' --“create projection sesson”s id
       and operator_name='DataTarget'
       and counter_name='written rows' 
       and counter_tag like '%isDV=f%'
       and eep.is_executing
       and counter_value != 0
       and eep.transaction_id=qp.transaction_id
       and eep.session_id= qp.session_id
       and qp.is_executing
       order by  counter_value desc
) a;

Leave a comment