The Effect of Conventional and Distal Radial Access Techniques on Radial Artery Structure and Vascular Functions
1Clinic of Cardiology, Muş State Hospital, Muş, Turkey
2Clinic of Cardiology, Ankara City Hospital, Ankara, Turkey
3Clinic of Radiology, Ankara City Hospital, Ankara, Turkey
4Department of Cardiology, Gazi University Faculty of Medicine, Ankara, Turkey
Keywords: Radial artery; coronary angiography; percutaneous coronary intervention
Introduction: To compare radial artery (RA) diameters, intima-media thickness (IMT), and vascular functions before and after conventional and distal RA cannulation. It has been proposed that distal transradial (DTRA) cannulation can reduce RA complications. However, there has been no comparative study examining the effects of DTR and conventional transradial (CTRA) intervention on RA structure and function.
Patients and Methods: Radial artery diameter, IMT, flow-mediated vasodilation (FMD) response were measured using conventional and Doppler RA ultrasonography before, and one day and one month after the procedure in patients who underwent CTRA or DTRA.
Results: While baseline and first-day IMT values were similar between the groups, first-month IMT values in the CTRA group were found to be significantly increased compared to those in the DTRA group (CTRA= 0.39 ± 0.10 mm, DTRA= 0.32 ± 0.07 mm, p= 0.016). While proximal RA occlusion developed in three patients who underwent CTRA, it did not develop in any of the patients undergoing DTRA (p= 0.072).
Conclusion: In patients who underwent CTRA, RA IMT increased significantly in the first month after the procedure, compared to patients who underwent DTRA. Proximal RAO was not observed in any of the patients who underwent DTRA.
Radial access in coronary angiographic procedures is recommended as a standard approach(1,2). However, as the number of transradial angiography (TRA) procedures increases, unique complications of radial angiography have been better understood. Previous studies have shown that radial artery (RA) diameter and flow-mediated vasodilation (FMD) response are decreased after TRA(3). RA occlusion (RAO) is another typical complication of TRA, and although it does not cause a serious loss of hand function, it may lead to symptoms such as pain in the wrist, cold thumb, and numbness(4). Furthermore, these complications may cause problems in the future if the RA is planned for a bypass graft or for an arteriovenous fistula.
In recent years, distal TRA (DTRA) has emerged as a new intervention method that can reduce the complications of RAs(5). Although there are many studies supporting DTRA is reliable and feasible, there are no studies examining the effect of distal RA intervention on RA diameter and endothelial functions compared to conventional TRA (CTRA) intervention. Our aim in this study was to investigate and compare the effects of cannulation via both techniques on RA diameter, endothelial functions, intima-media thickness (IMT), procedural success, and complications using conventional and Doppler ultrasonography (USG).
Patients and Methods
Between April and September 2019, male patients undergoing coronary angiography (CAG) with a diagnosis of acute coronary syndrome or stable coronary artery disease in the Ankara Numune Training and Research Hospital Cardiology Clinic, were included in our study. Informed consent was obtained from the patients and the study protocol was approved by the local ethics committee (E-19-2591). Fourty consecutive patients scheduled for CTRA and DTRA were included. Patients had similar demographics and laboratory parameters (Table 1). The exclusion criteria were as follows: patients who were < 18 or > 75 years old, female patients, patients who had a history of RA cannulation, whose Allen test was abnormal, who had peripheral artery disease, systemic inflammatory disease, chronic kidney failure, chronic liver disease or active infection. Patients’ age, comorbidities, medications before and after the angiography, and laboratory findings were recorded.
Radial Artery Ultrasonography Evaluation and Flow-Mediated Vasodilation Response
Radial artery conventional and Doppler USG examinations were performed the day before, the day after, and one month after the procedure. The examinations were carried out by a radiologist experienced in vascular USG, who used a linear array transducer (Epiq 5, Philips). Radial artery FMD response was evaluated as Corretti et al. reported previously(6). It has been shown that IMT is correlated with FMD, which is a good noninvasive indicator of endothelial function(7). After resting the patient in the supine position for 10 min, RA diameter was measured before vasodilation. The transducer was placed on the RA trace 3-4 cm proximal to the styloid process and set to give a display longitudinally in the region where the optimal image was obtained. RA diameter was measured intima to intima and recorded as the baseline value. Intima media thickness was measured by magnifying the image to show the lumen and internal elastic membrane. In the RA region where the diameter was measured, the blood flow velocity was measured with pulsed wave Doppler in peak systole. After recording the baseline RA diameter and flow rate, the cuff of a sphygmomanometer was attached to the patient’s arm. The cuff was inflated 50 mmHg above systolic blood pressure and was waited for ten minutes. Then the cuff was rapidly deflated, and the blood flow rate was measured within 15 s (a hyperemic response). A RA image was taken to evaluate FMD in response to this hyperemia. All measurements were made at the R peak according to electrocardiography. The maximum diameter (MaxD) from these measurements was used to calculate FMD, which was expressed as a percentage increase compared to the baseline vessel diameter (BD). Thus, the endothelium-dependent dilation was calculated via the equation FMD= [(MaxD − BD) / BD] × 100.
Arterial Cannulation and Coronary Angiography
Radial artery cannulations were performed by the same operator, who had experience with CTRA and DTRA techniques. All cannulations were performed from the right arm. In conventional RA cannulation, 1-2 mL of 2% lidocaine solution was applied under the skin for local anesthesia before entering the artery. The artery was punctured 1-2 cm proximally to the styloid process at an angle of 25-30° from the horizontal plane with an 18-21 G needle. A 6F transradial sheath (11 cm long 6F sheath and a 0.021-inch thick guidewire) was used. The guidewire was inserted through the needle after obtaining a pulsatile blood flow, indicating that we were in the artery. Then the sheath and the dilator were advanced over the wire. A cocktail of 200 μg nitroglycerin, 2.5 mg diltiazem, and 5000 IU unfractionated heparin was diluted to 20 cc serum physiologic for all patients. The dilator and wire were pulled back, and the cocktail was administered. The sheath was removed immediately after the procedure. Hemostasis was achieved by tight sponge roll compression. After approximately 30 min of tight compression, the tourniquet was loosened.
In distal RA cannulation, the right hand was placed in a position where the thumb could be grasped by the other four fingers and positioned parallel to the body at the patient’s hip. Then the RA was determined by following its trace in the snuffbox and was punctured with an 18-21 G needle at an angle of 30-45°. In the next stages, the same procedure was followed as for the CTRA. Diagnostic angiographies were performed with 6F Judkins left and right catheters. 6F guiding catheters were used in the case of angioplasty and stenting.
The data were analyzed with SPSS 22.00 statistical software. Distribution patterns of continuous variables were evaluated by the Shapiro-Wilk test. Categorical variables were expressed as numbers and ratios, and continuous variables as mean ± standard deviation values or median (and interquartile range, 25th-75th percentile) values according to the distribution pattern. Student’s t-test or Mann-Whitney U test was used to compare continuous variables between the CTRA and DTRA groups; Chi-square or Fisher test was used to compare categorical variables. Repeated RA and FMD measurements which were obtained before and after the angiography were analyzed using the Friedman test. Bonferroni correction was applied when making multiples comparisons. If a significant difference about the study endpoints was detected between the intervention site groups, ANCOVA analysis was performed to neutralize the possible confounders.
A total of 40 patients (20 DTRA-20 CTRA) were included in the study. The mean age, risk factors, and laboratory values before the procedure were similar in both groups (Table 1). Percutaneous coronary intervention was performed in 10 (50%) of the CTRA patients and 6 (30%) of the DTRA patients. Antiplatelet and anticoagulant use rates were similar in both groups before and after the procedure (Table 2). Median puncture attempts for successful RA cannulation were two in the CTRA group and five in the DTRA group (p< 0.001). Median sheath insertion times were 100 and 150 s in the CTRA and DTRA groups, respectively (p= 0.001).
• Radial Artery Diameters: Comparing the CTRA and DTRA groups in terms of RA diameter, the baseline, day one, and month one values were similar in both groups (Table 3). Repeated measurements within the groups revealed that the RA diameter increased significantly on day one and became similar to the baseline value at month one in both groups (Figure 1).
• Radial Artery Intima-Media Thicknesses: When the IMTs of the CTRA and DTRA groups were compared, the baseline and day one values were similar, while at month 1, those of the patients who underwent CTRA were significantly increased compared to the DTRA group (Table 3). When consecutive measurements obtained from the CTRA and DTRA groups were examined, the RA IMT values were similar in the group undergoing DTRA angiography before the procedure, and on the first day and first month after the procedure. In the CTRA group, IMT measured in the first month was increased significantly (Figure 2). CTRA was significantly associated with increased first month intima-media thickness. Although baseline intimamedia thickness has an effect on first month intima-media thickness, the effect of intervention type on first month radial artery intima-media thickness was independent from baseline radial artery intima-media thickness values (Table 4).
• Radial Artery FMD Response: When we look at the RA FMD response before the procedure, it was significantly lower in the DTRA group than in the CTRA group (Table 3). First-day and first-month FMD responses were similar in both groups. When the consecutive measurements obtained from the CTRA and DTRA groups were examined, it was observed that the FMD response decreased significantly on the first day but became similar to baseline values in the first month in both groups (Figure 3).
• Radial Artery Occlusion: Proximal RAO was observed in three patients in the CTRA group and none of the patients in the DTRA group (p= 0.072, Table 3), although distal RAO was observed in two DTRA patients. One patient with proximal RAO had pain at the intervention site while the others were asymptomatic.
In this study, the effects of distal vs. conventional transradial cannulation on RA diameters, IMT, and FMD response in patients undergoing CAG were investigated. For both cannulation types, RA diameter was found to be similar before, on the first day, and at month 1 after the procedure. Also, for both cannulation types, RA diameter increased significantly compared to its baseline value on the first day and then became similar to the baseline value in the first month. In terms of RA IMT, the values of both groups were similar at baseline and on day one, with the DTRA group also having similar values at month one. However the values of the CTRA group were higher at month one. While the FMD response was lower in the DTRA group at baseline than in the CTRA group, the responses were similar in both groups on the first day and in the first month after the procedure. It was observed that the FMD response decreased significantly on the first day after the procedure in both groups compared to baseline, and then returned to near baseline values in the first month. No RAO was observed in the DTRA group. To the best of our knowledge, this study is the first to examine structural and functional changes in RAs ultrasonographically in patients undergoing DTRA and CTRA.
The transradial approach was first used by Kiemeneij in 1993 for percutaneous coronary intervention(8). Since then, the popularity of transradial intervention has grown steadily, and is recommended with a class 1 indication radial access over femoral access, if done by an experienced radial interventionist(2). However, as experience with transradial techniques increases, the specific disadvantages and complications of this intervention method are becoming clearer. Various complications have been described in patients undergoing radial angiography, such as RA spasm, significant narrowing in the RA, RAO, hemorrhage at the intervention site, forearm hematoma, development of pseudoaneurysm at the intervention site, RA perforation and compartment syndrome of the forearm(9,10). In addition to these complications, studies are showing that dysfunction develops in the RA endothelium after radial intervention(10). Among these complications, RA stenosis, RAO, and RA endothelial dysfunction are closely related to damage caused by RA cannulation and compression applied for hemostasis after the procedure. Beyond the various symptoms caused by RAO, permanent RA dysfunction that develops after the procedure may limit the possibility of this vessel being used as a bypass conduit in the future or for creating an arteriovenous fistula in patients needing hemodialysis.
Distal transradial intervention might be a new alternative to prevent the aforementioned complications. Access to the distal TR artery from the radial fossa was first described by Babunashvili in two cases of the retrograde opening of occluded ipsilateral radial arteries(11). One of the advantages of distal, as opposed to conventional transradial cannulation, is that the RAO rates are exceptionally low, being one of the most common complications of CTRA procedures, occurring in 2.8- 11.7% of patients(12). RAO is usually asymptomatic and can be neglected due to the bilateral blood supply of the hand, but may rarely cause paresthesia and distal ischemia(13). However, even if the symptoms of RAO are ignored, the development of RAO should be prevented, as this RA segment may be used in coronary artery bypass grafting or for arterio-venous fistula in hemodialysis patients(14). Publications regarding DTRA presented quite rare RAO rates. While some series reported very rare RAO rates some publications reported no RAO after DTRA. Kaledin et al. reported that RAO occurred in the anatomic forearm in 0.4% of 1009 patients who underwent cannulation of the distal RA from the anatomical snuffbox. In 1.8% of these patients, RAO was observed at the snuffbox level. It has been reported that forearm RA flow continues actively in patients with snuffbox RAO(15). Oliveira et al. reported that distal and proximal RA pulses were palpable after DTRA in all patients discharged from the hospital(16). Also, in China, Yu et al. studied 92 patients who underwent DTRA and RAO was not detected in any of them(17). In our study, RAO was observed in the forearm in three of the patients who underwent CTRA, while RAO did not develop in the forearm in any of the patients who underwent DTRA. Distal RAO was observed in the snuffbox in two of the DTRA patients. Our findings seem to be compatible with the literature data.
Another parameter we examined in our study was the RA diameter. RA cannulation triggers inflammation by damaging the RA. This may cause a reduction in radial diameter in the long term. Several studies examined changes in RA diameter following conventional RA catheterization. In a study conducted by Nagai et al., the RA diameter was found to be lower than the baseline value on the second day and approximately three months later(9). In our study, while the RA diameters in both groups increased significantly on the first day after the procedure compared to the pre-procedure level, the RA diameters were similar to their baseline values after one month. The change in RA diameter was similar in both groups and, when compared to each other, the RA diameter was similar on the first day and first month. It is interesting to find that the RA diameter increased on the first day in our findings, contradicting the findings in the studies mentioned above; as many studies in the literature suggest that the RA diameter decreases even in the early period after RA cannulation.
However, some studies found the opposite. In an experimental study by Jamal et al., rabbit carotid arteries were damaged in different ways and how they responded to the damage was examined. They found that the resting artery diameter increased markedly for at least two days after damage caused by ballooning and excessive stretching. It was also found that there was no vasoconstrictor response in these arteries two days later(18). Again, in a study by Nakamura et al., the diameter of arteries increased significantly after laser iliac artery injury(19). In other words, after acute damage, arteries may enlarge, and our observations are consistent with these findings. In our study, it was observed that RA diameter tends towards its baseline level in the first month; two studies in the literature support our data in this regard(20,21). However, in a study conducted by Yoo et al., it was observed that the reduction of RA diameter with CTRA persisted for 4.5 months(22).
An important finding of our study is that the IMT of the CTRA group was significantly higher than the DTRA group in the first month. In the publications in which the RA structure was evaluated by intravascular USG, it was observed that the reduction in RA diameter was particularly related to intimal thickening, which is shown to be closely related to recurrent TRACAG(9-22). Another study concluded that radial intervention reduces radial graft patency and causes intimal hyperplasia in the artery(20). Diameter reduction and intimal thickening pose a great risk for dysfunction of the RA graft to be used for coronary artery bypass grafting surgery. The finding of intimal thickening in the CTRA intervention arm in our study is supported by previous studies(23,24). Intimal thickening was shown to be one of the main causes of diameter reduction in the subsequent period. It is noteworthy that there is no significant increase in intimal thickening in the DTRA arm; this is reported for the first time in the literature with the present study.
In our study, we applied an FMD test to evaluate the endothelial function of the RA. We determined that there was a significant decrease, compared to the baseline values, in both intervention arms on the first day. Although no significant difference was found in the CTRA intervention arm between the baseline and month one values, the day one values were found to be lower, similar to the baseline values in the DTRA intervention arm. The FMD test is the most commonly accepted non-invasive test that shows arterial endothelial function. In a study involving 30 patients who underwent CAG via an RA, the RAs were examined ultrasonographically 12 months (range, 10-14 months) after the procedure, and it was concluded that FMD was preserved while a significant reduction in the RA diameter was detected(25). In a meta-analysis involving 12 randomized controlled trials, there was a marked decrease in endothelial-dependent response and a nonsignificant decrease in endothelial-independent vasomotor function after RA catheterization (>1 month). The findings of a recent meta-analysis showed permanent impairment in endothelial function, which continued at least a few months after catheterization(26). Since the RA is a muscular artery, it is prone to vasospasm. The longterm patency of the graft used in coronary artery bypass surgery depends on normal endothelial function(27,28). Therefore, the normalization of FMD after a radial intervention is of great importance in the long run. CTRA-CAG impairs the endothelial function of the RA in the short and long term and reduces its suitability for coronary artery bypass grafting surgery(29).
The most important limitation of the study is the low number of patients; the power of this study is not sufficient to evaluate the differences in variables such as RAO and RA diameter. However, the study reached sufficient power for the evaluation of IMT, and a significant difference was found between the two groups. In addition, the successive examination of the patients’ variables before, on the first day after, and one month after the procedure is valuable in terms of showing the temporal change of the effects on the RA in both cannulation types. Other limitations include being a single-center study and having no validation of USG data and no reproducibility. Differences in medical treatment may affect vascular functions but, in our study, we believe this had little effect because both groups had similar rates of coronary artery disease and medical treatment.
In patients who underwent CTRA, RA IMT increased significantly in the first month after the procedure compared to those who underwent DTRA. Flow-mediated dilation responses were similar in DTRA and CTRA patients one day and one month after the procedure. In both cannulation types, the FMD response decreased significantly on the first day after the procedure, but this decrease disappeared during the first month. Proximal RAO was not observed in any patients who received DTRA.
The approval for this study obtained from SBÜ Ankara Numune Training and Research Hospital, Clinical Research Ethics Committee (Decision No: E-19-2591, Date: 28.03.2019).
This is retrospective study, we could not obtain written informed consent from the participants.
Concept/Design - CÖ, EÖ; Analysis/Interpretation - EK; Data Collection - İEY, VÇ; Writing - CÖ, EK, MÇ; Critical Revision - AK, EÖ; Final Approval - CÖ, EK, EÖ; Statistical Analysis - BS, AK; Overall Responsibility - CÖ.
The authors declared that there was no conflict of interest during the preparation and publication of this article.
The authors declared that this study has received no financial support.
- Ibanez B, James S, Agewall S, Antunes MJ, Bucciarelli-Ducci C, Bueno H, et al. 2017 ESC Guidelines for the management of acute myocardial infarction in patients presenting with ST-segment elevation: The task force for the management of acute myocardial infarction in patients presenting with ST-segment elevation of the European Society of Cardiology (ESC). Eur Heart J 2017;39(2):119-77. [Crossref]
- Neumann FJ, Sousa-Uva M, Ahlsson A, Alfonso F, Banning AP, Benedetto U, et al. 2018 ESC/EACTS guidelines on myocardial revascularization. Eur Hearth J 2018;40(2):87-165. [Crossref]
- Burstein JM, Gidrewicz D, Hutchison SJ, Holmes K, Jolly S, Cantor WJ. Impact of radial artery cannulation for coronary angiography and angioplasty on radial artery function. Am J Card 2007;99(4):457-9. [Crossref]
- Rhyne D, Mann T. Hand ischemia resulting from a transradial intervention: successful management with radial artery angioplasty. Catheter Cardiovasc Interv 2010;76(3):383-6. [Crossref]
- Al-Azizi KM, Lotfi AS. The distal left radial artery access for coronary angiography and intervention: a new era. Cardiovasc Revasc Med 2018;19(8):35-40. [Crossref]
- Corretti MC, Anderson TJ, Benjamin EJ, Celermajer D, Charbonneau F, Creager MA, et al. Guidelines for the ultrasound assessment of endothelial-dependent flow-mediated vasodilation of the brachial artery: a report of the international brachial artery reactivity task force. J Am Coll Cardiol 2002;39(2):257-65. [Crossref]
- Korkmaz H, Akbulut M, Özbay Y, Koç M. Brakiyal arter intima-media kalınlığının endotel fonksiyonu ve sol ventrikül kütle indeksi ile ilişkisi. Anatol J Cardiol. 2010;10(3).
- Kiemeneij F, Jan Laarman G. Percutaneous transradial artery approach for coronary stent implantation. Catheter Cardiovasc Diagnosis 1993;30(2):173-8. [Crossref]
- Nagai S, Abe S, Sato T, Hozawa K, Yuki K, Hanashima K, et al. Ultrasonic assessment of vascular complications in coronary angiography and angioplasty after transradial approach. Catheter Cardiovasc Interv 1999;83(2):180-6. [Crossref]
- Burzotta F, Trani C, Mazzari MA, Tommasino A, Niccoli G, Porto I, et al. Vascular complications and access crossover in 10.676 transradial percutaneous coronary procedures. Am Heart J 2012;163(2):230-8. [Crossref]
- Babunashvili A, Dundua D. Recanalization and reuse of early occluded radial artery within 6 days after previous transradial diagnostic procedure. Catheter Cardiovasc Interv 2011;77(4):530-6. [Crossref]
- Chen Y, Ke Z, Xiao J, Lin M, Huang X, Yan C, et al. Subcutaneous injection of nitroglycerin at the radial artery puncture site reduces the risk of early radial artery occlusion after transradial coronary catheterization: a randomized, placebo-controlled clinical trial. Circ Cardiovasc Interv 2018;11(7):e006571. [Crossref]
- Wagener JF, Rao SV. Radial artery occlusion after transradial approach to cardiac catheterization. Curr Atheroscler Rep 2015;17(3):9. [Crossref]
- Shemesh D, Goldin I, Verstandig A, Berelowitz D, Zaghal I, Olsha O. Upper limb grafts for hemodialysis access. J Vasc Access 2015;16(Suppl-9):34-9. [Crossref]
- Kaledin A, Kochanov I, Podmetin P, Seletsky S, Ardeev V. Distal radial artery in endovascular interventions. Distal radial artery in endovascular interventions, 2017. Available from: https://www.researchgate.net/publication/319162208_Distal_radial_artery_in_endovascular_interventions
- Oliveira MDP, Navarro EC, Kiemeneij F. Distal transradial access as default approach for coronary angiography and interventions. CDT 2019;9(5):513. [Crossref]
- Yu W, Hu P, Wang S, Yao L, Wang H, Dou L, et al. Distal radial artery access in the anatomical snuffbox for coronary angiography and intervention: A single center experience. Med 2020;99(3):e18330. [Crossref]
- Jamal A, Bendeck M, Langille B. Structural changes and recovery of function after arterial injury. Arterioscler Thromb 1992;12(3):307-17. [Crossref]
- Nakamura F, Kvasnicka J, Levame M, Lange F, Bousbaa H, Geschwind HJ. Acute response of the arterial wall to pulsed laser irradiation. Lasers Surg Med 1993;13(4):412-20. [Crossref]
- Kamiya H, Ushijima T, Kanamori T, Ikeda C, Nakagaki C, Ueyama K, et al. Use of the radial artery graft after transradial catheterization: is it suitable as a bypass conduit? The Ann Thorac Surg 2003;76(5):1505-9. [Crossref]
- Park KH, Park WJ, Kim MK, Park DW, Park JH, Kim HS, et al. Effects of trimetazidine on endothelial dysfunction after sheath injury of radial artery. Am J Cardiol 2010;105(12):1723-7. [Crossref]
- Yoo BS, Lee SH, Ko JY, Lee BK, Kim SN, Lee MO, et al. Procedural outcomes of repeated transradial coronary procedure. Catheter Cardiovasc Interv 2003;58(3):301-4. [Crossref]
- Yonetsu T, Kakuta T, Lee T, Takayama K, Kakita K, Iwamoto T, et al. Assessment of acute injuries and chronic intimal thickening of the radial artery after transradial coronary intervention by optical coherence tomography. Eur Heart J 2010;31(13):1608-15. [Crossref]
- Wakeyama T, Ogawa H, Iida H, Takaki A, Iwami T, Mochizuki M, et al. Intima-media thickening of the radial artery after transradial intervention: an intravascular ultrasound study. J Am Coll Cardiol 2003;41(7):1109-14. [Crossref]
- Madssen E, Haere P, Wiseth R. Radial artery diameter and vasodilatory properties after transradial coronary angiography. Ann Thorac Surg 2006;82(5):1698-702. [Crossref]
- Antonopoulos AS, Latsios G, Oikonomou E, Aznaouridis K, Papanikolaou A, Syrseloudis D, et al. Long-term endothelial dysfunction after trans-radial catheterization: A meta-analytic approach. J Card Surg 2017;32(8):464-73. [Crossref]
- Moran SV, Baeza R, Guarda E, Zalaquett R, Irarrazaval MJ, Marchant E, et al. Predictors of radial artery patency for coronary bypass operations. Ann Thorac Surg 2001;72(5):1552-6. [Crossref]
- He GW. Arterial grafts for coronary surgery: vasospasm and patency rate. J Thorac Cardiovasc Surg 2001;121(3):431-3. [Crossref]
- Yan Z, Zhou Y, Zhao Y, Zhou Z, Yang S, Wang Z. Impact of transradial coronary procedures on radial artery. Angiology 2010;61(1):8-13. [Crossref]